Jul 30 19:22:32.473037 ip-10-0-129-105 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 30 19:22:32.473046 ip-10-0-129-105 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 30 19:22:32.473053 ip-10-0-129-105 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 30 19:22:32.473294 ip-10-0-129-105 systemd[1]: Failed to start Kubernetes Kubelet. Jul 30 19:22:42.478244 ip-10-0-129-105 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 30 19:22:42.478269 ip-10-0-129-105 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 5de45d0f52e74a6796e87756b5c5dd42 -- Jul 30 19:24:37.665634 ip-10-0-129-105 systemd[1]: Starting Kubernetes Kubelet... Jul 30 19:24:38.084892 ip-10-0-129-105 kubenswrapper[2556]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 30 19:24:38.084892 ip-10-0-129-105 kubenswrapper[2556]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 30 19:24:38.084892 ip-10-0-129-105 kubenswrapper[2556]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 30 19:24:38.084892 ip-10-0-129-105 kubenswrapper[2556]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 30 19:24:38.084892 ip-10-0-129-105 kubenswrapper[2556]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 30 19:24:38.085824 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.085754 2556 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 30 19:24:38.089582 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089568 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 19:24:38.089582 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089582 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 19:24:38.089642 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089585 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 19:24:38.089642 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089589 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 19:24:38.089642 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089593 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 19:24:38.089642 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089596 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 19:24:38.089642 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089598 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 19:24:38.089642 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089601 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 19:24:38.089642 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089603 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 19:24:38.089642 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089606 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 19:24:38.089642 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089609 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 19:24:38.089642 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089611 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 19:24:38.089642 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089614 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 19:24:38.089642 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089616 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 19:24:38.089642 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089619 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 19:24:38.089642 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089621 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 19:24:38.089642 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089624 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 19:24:38.089642 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089633 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 19:24:38.089642 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089636 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 19:24:38.089642 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089638 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 19:24:38.089642 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089641 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 19:24:38.090177 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089643 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 19:24:38.090177 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089647 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 19:24:38.090177 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089649 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 19:24:38.090177 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089654 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 19:24:38.090177 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089658 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 19:24:38.090177 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089661 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 19:24:38.090177 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089664 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 19:24:38.090177 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089666 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 19:24:38.090177 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089669 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 19:24:38.090177 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089672 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 19:24:38.090177 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089675 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 19:24:38.090177 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089677 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 19:24:38.090177 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089680 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 19:24:38.090177 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089682 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 19:24:38.090177 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089686 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 19:24:38.090177 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089688 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 19:24:38.090177 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089691 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 19:24:38.090177 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089693 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 19:24:38.090177 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089696 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 19:24:38.090848 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089698 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 19:24:38.090848 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089700 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 19:24:38.090848 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089703 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 19:24:38.090848 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089706 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 19:24:38.090848 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089708 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 19:24:38.090848 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089712 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 19:24:38.090848 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089714 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 19:24:38.090848 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089716 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 19:24:38.090848 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089719 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 30 19:24:38.090848 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089721 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 19:24:38.090848 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089723 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 19:24:38.090848 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089726 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 19:24:38.090848 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089728 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 19:24:38.090848 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089731 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 19:24:38.090848 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089734 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 19:24:38.090848 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089737 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 19:24:38.090848 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089740 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 19:24:38.090848 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089742 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 19:24:38.090848 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089745 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 19:24:38.090848 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089747 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 19:24:38.091328 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089749 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 19:24:38.091328 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089752 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 19:24:38.091328 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089755 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 19:24:38.091328 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089758 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 19:24:38.091328 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089760 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 19:24:38.091328 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089763 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 19:24:38.091328 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089766 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 19:24:38.091328 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089768 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 19:24:38.091328 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089770 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 19:24:38.091328 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089773 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 19:24:38.091328 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089775 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 19:24:38.091328 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089777 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 19:24:38.091328 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089780 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 19:24:38.091328 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089782 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 19:24:38.091328 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089785 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 19:24:38.091328 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089787 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 19:24:38.091328 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089789 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 19:24:38.091328 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089792 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 19:24:38.091328 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089794 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 19:24:38.091328 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089797 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 19:24:38.091807 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089800 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 19:24:38.091807 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089804 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 19:24:38.091807 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089806 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 19:24:38.091807 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089809 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 19:24:38.091807 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089812 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 19:24:38.091807 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089815 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 19:24:38.091807 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089818 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 19:24:38.091807 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089820 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 19:24:38.091807 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089823 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 19:24:38.091807 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089826 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 19:24:38.091807 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089828 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 19:24:38.091807 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089831 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 19:24:38.091807 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089833 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 19:24:38.091807 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089837 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 19:24:38.091807 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089840 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 19:24:38.091807 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089843 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 19:24:38.091807 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089846 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 19:24:38.091807 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089848 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 19:24:38.091807 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.089851 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 19:24:38.092587 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092572 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 19:24:38.092587 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092586 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 19:24:38.092648 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092589 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 19:24:38.092648 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092592 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 19:24:38.092648 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092595 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 19:24:38.092648 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092598 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 19:24:38.092648 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092602 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 19:24:38.092648 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092605 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 19:24:38.092648 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092609 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 19:24:38.092648 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092612 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 19:24:38.092648 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092615 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 19:24:38.092648 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092618 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 19:24:38.092648 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092620 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 19:24:38.092648 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092623 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 19:24:38.092648 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092626 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 19:24:38.092648 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092629 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 19:24:38.092648 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092631 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 19:24:38.092648 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092634 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 19:24:38.092648 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092636 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 19:24:38.092648 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092639 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 19:24:38.092648 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092642 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 19:24:38.092648 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092644 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 19:24:38.093133 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092647 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 19:24:38.093133 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092649 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 19:24:38.093133 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092659 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 19:24:38.093133 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092662 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 19:24:38.093133 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092664 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 19:24:38.093133 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092667 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 19:24:38.093133 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092669 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 19:24:38.093133 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092676 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 19:24:38.093133 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092678 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 19:24:38.093133 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092681 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 19:24:38.093133 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092685 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 19:24:38.093133 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092688 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 19:24:38.093133 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092691 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 19:24:38.093133 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092694 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 19:24:38.093133 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092696 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 19:24:38.093133 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092699 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 19:24:38.093133 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092701 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 19:24:38.093133 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092704 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 19:24:38.093133 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092706 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 19:24:38.093593 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092708 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 19:24:38.093593 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092711 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 19:24:38.093593 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092713 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 19:24:38.093593 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092716 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 19:24:38.093593 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092718 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 19:24:38.093593 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092720 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 19:24:38.093593 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092723 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 19:24:38.093593 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092725 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 19:24:38.093593 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092728 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 19:24:38.093593 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092730 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 19:24:38.093593 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092734 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 19:24:38.093593 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092737 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 19:24:38.093593 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092740 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 19:24:38.093593 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092743 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 19:24:38.093593 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092747 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 19:24:38.093593 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092750 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 19:24:38.093593 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092753 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 19:24:38.093593 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092756 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 19:24:38.093593 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092758 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 19:24:38.093593 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092761 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 19:24:38.094092 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092765 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 19:24:38.094092 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092769 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 19:24:38.094092 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092771 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 19:24:38.094092 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092774 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 19:24:38.094092 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092776 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 19:24:38.094092 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092779 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 19:24:38.094092 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092781 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 19:24:38.094092 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092784 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 19:24:38.094092 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092786 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 19:24:38.094092 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092788 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 19:24:38.094092 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092791 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 19:24:38.094092 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092793 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 19:24:38.094092 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092796 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 19:24:38.094092 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092798 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 19:24:38.094092 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092800 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 19:24:38.094092 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092803 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 19:24:38.094092 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092805 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 19:24:38.094092 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092808 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 19:24:38.094092 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092810 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 19:24:38.094092 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092813 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 19:24:38.094576 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092816 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 19:24:38.094576 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092818 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 19:24:38.094576 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092822 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 19:24:38.094576 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092824 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 19:24:38.094576 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092826 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 19:24:38.094576 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092829 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 19:24:38.094576 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092833 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 19:24:38.094576 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092835 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 19:24:38.094576 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092838 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 19:24:38.094576 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092840 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 30 19:24:38.094576 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092843 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 19:24:38.094576 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092846 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 19:24:38.094576 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092849 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 19:24:38.094576 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092851 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 19:24:38.094576 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092853 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 19:24:38.094576 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092856 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 19:24:38.094576 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092858 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 19:24:38.094576 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.092861 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 19:24:38.094576 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093196 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 19:24:38.094576 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093200 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 19:24:38.095079 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093203 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 19:24:38.095079 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093205 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 19:24:38.095079 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093208 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 19:24:38.095079 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093211 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 19:24:38.095079 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093213 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 19:24:38.095079 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093215 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 19:24:38.095079 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093218 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 19:24:38.095079 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093220 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 19:24:38.095079 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093223 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 19:24:38.095079 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093225 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 19:24:38.095079 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093228 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 19:24:38.095079 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093230 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 19:24:38.095079 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093233 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 19:24:38.095079 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093235 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 19:24:38.095079 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093240 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 19:24:38.095079 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093244 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 19:24:38.095079 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093247 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 19:24:38.095079 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093250 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 19:24:38.095079 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093254 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 19:24:38.095547 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093257 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 19:24:38.095547 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093260 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 19:24:38.095547 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093263 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 19:24:38.095547 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093265 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 19:24:38.095547 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093268 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 19:24:38.095547 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093270 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 19:24:38.095547 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093273 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 19:24:38.095547 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093282 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 19:24:38.095547 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093285 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 19:24:38.095547 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093287 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 19:24:38.095547 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093290 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 19:24:38.095547 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093292 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 19:24:38.095547 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093295 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 19:24:38.095547 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093297 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 19:24:38.095547 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093299 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 19:24:38.095547 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093302 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 19:24:38.095547 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093304 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 19:24:38.095547 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093306 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 19:24:38.095547 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093309 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 19:24:38.095991 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093311 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 19:24:38.095991 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093314 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 19:24:38.095991 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093316 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 19:24:38.095991 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093318 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 19:24:38.095991 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093321 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 19:24:38.095991 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093323 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 19:24:38.095991 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093326 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 19:24:38.095991 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093328 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 19:24:38.095991 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093330 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 30 19:24:38.095991 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093333 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 19:24:38.095991 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093336 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 19:24:38.095991 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093338 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 19:24:38.095991 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093341 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 19:24:38.095991 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093344 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 19:24:38.095991 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093346 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 19:24:38.095991 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093349 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 19:24:38.095991 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093351 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 19:24:38.095991 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093354 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 19:24:38.095991 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093356 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 19:24:38.096480 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093358 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 19:24:38.096480 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093361 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 19:24:38.096480 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093364 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 19:24:38.096480 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093366 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 19:24:38.096480 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093368 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 19:24:38.096480 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093371 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 19:24:38.096480 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093373 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 19:24:38.096480 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093376 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 19:24:38.096480 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093378 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 19:24:38.096480 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093397 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 19:24:38.096480 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093401 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 19:24:38.096480 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093405 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 19:24:38.096480 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093410 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 19:24:38.096480 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093413 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 19:24:38.096480 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093415 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 19:24:38.096480 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093417 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 19:24:38.096480 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093420 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 19:24:38.096480 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093422 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 19:24:38.096480 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093425 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 19:24:38.096480 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093428 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 19:24:38.096955 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093430 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 19:24:38.096955 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093434 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 19:24:38.096955 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093436 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 19:24:38.096955 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093440 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 19:24:38.096955 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093443 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 19:24:38.096955 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093445 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 19:24:38.096955 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093447 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 19:24:38.096955 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093450 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 19:24:38.096955 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093452 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 19:24:38.096955 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093455 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 19:24:38.096955 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093457 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 19:24:38.096955 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093460 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 19:24:38.096955 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093463 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 19:24:38.096955 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093465 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 19:24:38.096955 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093469 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 19:24:38.096955 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093473 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 19:24:38.096955 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093476 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 19:24:38.096955 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093479 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 19:24:38.096955 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093481 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 19:24:38.096955 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.093484 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 19:24:38.097423 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093555 2556 flags.go:64] FLAG: --address="0.0.0.0" Jul 30 19:24:38.097423 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093563 2556 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 30 19:24:38.097423 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093572 2556 flags.go:64] FLAG: --anonymous-auth="true" Jul 30 19:24:38.097423 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093579 2556 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 30 19:24:38.097423 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093585 2556 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 30 19:24:38.097423 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093589 2556 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 30 19:24:38.097423 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093593 2556 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 30 19:24:38.097423 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093597 2556 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 30 19:24:38.097423 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093600 2556 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 30 19:24:38.097423 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093603 2556 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 30 19:24:38.097423 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093607 2556 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 30 19:24:38.097423 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093610 2556 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 30 19:24:38.097423 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093613 2556 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 30 19:24:38.097423 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093616 2556 flags.go:64] FLAG: --cgroup-root="" Jul 30 19:24:38.097423 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093618 2556 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 30 19:24:38.097423 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093622 2556 flags.go:64] FLAG: --client-ca-file="" Jul 30 19:24:38.097423 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093626 2556 flags.go:64] FLAG: --cloud-provider="external" Jul 30 19:24:38.097423 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093629 2556 flags.go:64] FLAG: --cluster-dns="[]" Jul 30 19:24:38.097423 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093633 2556 flags.go:64] FLAG: --cluster-domain="" Jul 30 19:24:38.097423 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093636 2556 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 30 19:24:38.097423 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093639 2556 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 30 19:24:38.097423 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093642 2556 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 30 19:24:38.097423 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093645 2556 flags.go:64] FLAG: --container-log-max-files="5" Jul 30 19:24:38.097423 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093649 2556 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 30 19:24:38.098013 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093652 2556 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 30 19:24:38.098013 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093655 2556 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 30 19:24:38.098013 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093658 2556 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 30 19:24:38.098013 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093661 2556 flags.go:64] FLAG: --contention-profiling="false" Jul 30 19:24:38.098013 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093664 2556 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 30 19:24:38.098013 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093667 2556 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 30 19:24:38.098013 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093670 2556 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 30 19:24:38.098013 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093673 2556 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 30 19:24:38.098013 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093676 2556 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 30 19:24:38.098013 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093679 2556 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 30 19:24:38.098013 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093682 2556 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 30 19:24:38.098013 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093687 2556 flags.go:64] FLAG: --enable-load-reader="false" Jul 30 19:24:38.098013 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093689 2556 flags.go:64] FLAG: --enable-server="true" Jul 30 19:24:38.098013 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093692 2556 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 30 19:24:38.098013 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093697 2556 flags.go:64] FLAG: --event-burst="100" Jul 30 19:24:38.098013 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093700 2556 flags.go:64] FLAG: --event-qps="50" Jul 30 19:24:38.098013 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093703 2556 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 30 19:24:38.098013 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093706 2556 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 30 19:24:38.098013 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093708 2556 flags.go:64] FLAG: --eviction-hard="" Jul 30 19:24:38.098013 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093712 2556 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 30 19:24:38.098013 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093714 2556 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 30 19:24:38.098013 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093717 2556 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 30 19:24:38.098013 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093720 2556 flags.go:64] FLAG: --eviction-soft="" Jul 30 19:24:38.098013 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093723 2556 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 30 19:24:38.098013 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093726 2556 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 30 19:24:38.098637 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093729 2556 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 30 19:24:38.098637 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093732 2556 flags.go:64] FLAG: --experimental-mounter-path="" Jul 30 19:24:38.098637 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093735 2556 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 30 19:24:38.098637 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093738 2556 flags.go:64] FLAG: --fail-swap-on="true" Jul 30 19:24:38.098637 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093740 2556 flags.go:64] FLAG: --feature-gates="" Jul 30 19:24:38.098637 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093744 2556 flags.go:64] FLAG: --file-check-frequency="20s" Jul 30 19:24:38.098637 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093747 2556 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 30 19:24:38.098637 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093750 2556 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 30 19:24:38.098637 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093752 2556 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 30 19:24:38.098637 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093755 2556 flags.go:64] FLAG: --healthz-port="10248" Jul 30 19:24:38.098637 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093758 2556 flags.go:64] FLAG: --help="false" Jul 30 19:24:38.098637 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093760 2556 flags.go:64] FLAG: --hostname-override="ip-10-0-129-105.ec2.internal" Jul 30 19:24:38.098637 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093763 2556 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 30 19:24:38.098637 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093766 2556 flags.go:64] FLAG: --http-check-frequency="20s" Jul 30 19:24:38.098637 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093770 2556 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 30 19:24:38.098637 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093774 2556 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 30 19:24:38.098637 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093777 2556 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 30 19:24:38.098637 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093780 2556 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 30 19:24:38.098637 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093784 2556 flags.go:64] FLAG: --image-service-endpoint="" Jul 30 19:24:38.098637 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093787 2556 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 30 19:24:38.098637 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093789 2556 flags.go:64] FLAG: --kube-api-burst="100" Jul 30 19:24:38.098637 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093792 2556 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 30 19:24:38.098637 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093795 2556 flags.go:64] FLAG: --kube-api-qps="50" Jul 30 19:24:38.098637 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093798 2556 flags.go:64] FLAG: --kube-reserved="" Jul 30 19:24:38.099221 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093801 2556 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 30 19:24:38.099221 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093804 2556 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 30 19:24:38.099221 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093806 2556 flags.go:64] FLAG: --kubelet-cgroups="" Jul 30 19:24:38.099221 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093810 2556 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 30 19:24:38.099221 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093812 2556 flags.go:64] FLAG: --lock-file="" Jul 30 19:24:38.099221 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093815 2556 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 30 19:24:38.099221 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093818 2556 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 30 19:24:38.099221 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093820 2556 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 30 19:24:38.099221 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093826 2556 flags.go:64] FLAG: --log-json-split-stream="false" Jul 30 19:24:38.099221 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093829 2556 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 30 19:24:38.099221 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093832 2556 flags.go:64] FLAG: --log-text-split-stream="false" Jul 30 19:24:38.099221 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093834 2556 flags.go:64] FLAG: --logging-format="text" Jul 30 19:24:38.099221 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093837 2556 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 30 19:24:38.099221 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093840 2556 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 30 19:24:38.099221 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093843 2556 flags.go:64] FLAG: --manifest-url="" Jul 30 19:24:38.099221 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093846 2556 flags.go:64] FLAG: --manifest-url-header="" Jul 30 19:24:38.099221 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093849 2556 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 30 19:24:38.099221 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093852 2556 flags.go:64] FLAG: --max-open-files="1000000" Jul 30 19:24:38.099221 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093856 2556 flags.go:64] FLAG: --max-pods="110" Jul 30 19:24:38.099221 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093859 2556 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 30 19:24:38.099221 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093862 2556 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 30 19:24:38.099221 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093864 2556 flags.go:64] FLAG: --memory-manager-policy="None" Jul 30 19:24:38.099221 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093869 2556 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 30 19:24:38.099221 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093872 2556 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 30 19:24:38.099221 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093875 2556 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 30 19:24:38.099819 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093878 2556 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 30 19:24:38.099819 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093887 2556 flags.go:64] FLAG: --node-status-max-images="50" Jul 30 19:24:38.099819 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093890 2556 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 30 19:24:38.099819 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093893 2556 flags.go:64] FLAG: --oom-score-adj="-999" Jul 30 19:24:38.099819 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093895 2556 flags.go:64] FLAG: --pod-cidr="" Jul 30 19:24:38.099819 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093898 2556 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8" Jul 30 19:24:38.099819 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093904 2556 flags.go:64] FLAG: --pod-manifest-path="" Jul 30 19:24:38.099819 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093906 2556 flags.go:64] FLAG: --pod-max-pids="-1" Jul 30 19:24:38.099819 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093909 2556 flags.go:64] FLAG: --pods-per-core="0" Jul 30 19:24:38.099819 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093912 2556 flags.go:64] FLAG: --port="10250" Jul 30 19:24:38.099819 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093915 2556 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 30 19:24:38.099819 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093918 2556 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ff2c8d6d6210c39d" Jul 30 19:24:38.099819 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093921 2556 flags.go:64] FLAG: --qos-reserved="" Jul 30 19:24:38.099819 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093924 2556 flags.go:64] FLAG: --read-only-port="10255" Jul 30 19:24:38.099819 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093926 2556 flags.go:64] FLAG: --register-node="true" Jul 30 19:24:38.099819 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093929 2556 flags.go:64] FLAG: --register-with-taints="" Jul 30 19:24:38.099819 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093933 2556 flags.go:64] FLAG: --registry-burst="10" Jul 30 19:24:38.099819 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093942 2556 flags.go:64] FLAG: --registry-qps="5" Jul 30 19:24:38.099819 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093945 2556 flags.go:64] FLAG: --reserved-cpus="" Jul 30 19:24:38.099819 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093947 2556 flags.go:64] FLAG: --reserved-memory="" Jul 30 19:24:38.099819 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093951 2556 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 30 19:24:38.099819 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093954 2556 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 30 19:24:38.099819 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093956 2556 flags.go:64] FLAG: --rotate-certificates="false" Jul 30 19:24:38.099819 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093959 2556 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 30 19:24:38.099819 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093962 2556 flags.go:64] FLAG: --runonce="false" Jul 30 19:24:38.100411 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093964 2556 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 30 19:24:38.100411 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093967 2556 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 30 19:24:38.100411 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093970 2556 flags.go:64] FLAG: --seccomp-default="false" Jul 30 19:24:38.100411 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093973 2556 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 30 19:24:38.100411 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093978 2556 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 30 19:24:38.100411 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093981 2556 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 30 19:24:38.100411 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093983 2556 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 30 19:24:38.100411 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093986 2556 flags.go:64] FLAG: --storage-driver-password="root" Jul 30 19:24:38.100411 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093989 2556 flags.go:64] FLAG: --storage-driver-secure="false" Jul 30 19:24:38.100411 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093993 2556 flags.go:64] FLAG: --storage-driver-table="stats" Jul 30 19:24:38.100411 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093996 2556 flags.go:64] FLAG: --storage-driver-user="root" Jul 30 19:24:38.100411 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.093998 2556 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 30 19:24:38.100411 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.094001 2556 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 30 19:24:38.100411 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.094004 2556 flags.go:64] FLAG: --system-cgroups="" Jul 30 19:24:38.100411 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.094007 2556 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 30 19:24:38.100411 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.094012 2556 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 30 19:24:38.100411 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.094014 2556 flags.go:64] FLAG: --tls-cert-file="" Jul 30 19:24:38.100411 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.094017 2556 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 30 19:24:38.100411 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.094021 2556 flags.go:64] FLAG: --tls-min-version="" Jul 30 19:24:38.100411 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.094024 2556 flags.go:64] FLAG: --tls-private-key-file="" Jul 30 19:24:38.100411 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.094026 2556 flags.go:64] FLAG: --topology-manager-policy="none" Jul 30 19:24:38.100411 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.094029 2556 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 30 19:24:38.100411 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.094032 2556 flags.go:64] FLAG: --topology-manager-scope="container" Jul 30 19:24:38.100411 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.094035 2556 flags.go:64] FLAG: --v="2" Jul 30 19:24:38.100411 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.094038 2556 flags.go:64] FLAG: --version="false" Jul 30 19:24:38.101021 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.094048 2556 flags.go:64] FLAG: --vmodule="" Jul 30 19:24:38.101021 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.094051 2556 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 30 19:24:38.101021 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.094054 2556 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 30 19:24:38.101021 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094159 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 19:24:38.101021 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094163 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 19:24:38.101021 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094166 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 19:24:38.101021 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094169 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 19:24:38.101021 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094171 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 19:24:38.101021 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094174 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 19:24:38.101021 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094177 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 19:24:38.101021 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094179 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 19:24:38.101021 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094183 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 19:24:38.101021 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094186 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 19:24:38.101021 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094189 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 19:24:38.101021 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094191 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 19:24:38.101021 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094194 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 19:24:38.101021 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094198 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 19:24:38.101021 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094200 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 19:24:38.101021 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094203 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 30 19:24:38.101021 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094205 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 19:24:38.101499 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094208 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 19:24:38.101499 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094210 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 19:24:38.101499 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094215 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 19:24:38.101499 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094219 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 19:24:38.101499 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094223 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 19:24:38.101499 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094226 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 19:24:38.101499 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094229 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 19:24:38.101499 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094232 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 19:24:38.101499 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094235 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 19:24:38.101499 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094237 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 19:24:38.101499 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094240 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 19:24:38.101499 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094243 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 19:24:38.101499 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094245 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 19:24:38.101499 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094248 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 19:24:38.101499 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094251 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 19:24:38.101499 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094253 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 19:24:38.101499 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094256 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 19:24:38.101499 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094259 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 19:24:38.101499 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094261 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 19:24:38.101953 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094264 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 19:24:38.101953 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094266 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 19:24:38.101953 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094269 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 19:24:38.101953 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094271 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 19:24:38.101953 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094275 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 19:24:38.101953 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094278 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 19:24:38.101953 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094280 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 19:24:38.101953 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094282 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 19:24:38.101953 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094285 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 19:24:38.101953 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094289 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 19:24:38.101953 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094292 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 19:24:38.101953 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094295 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 19:24:38.101953 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094297 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 19:24:38.101953 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094300 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 19:24:38.101953 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094302 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 19:24:38.101953 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094305 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 19:24:38.101953 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094307 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 19:24:38.101953 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094310 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 19:24:38.101953 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094312 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 19:24:38.101953 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094315 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 19:24:38.102441 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094318 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 19:24:38.102441 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094320 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 19:24:38.102441 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094323 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 19:24:38.102441 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094325 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 19:24:38.102441 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094328 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 19:24:38.102441 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094330 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 19:24:38.102441 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094333 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 19:24:38.102441 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094335 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 19:24:38.102441 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094338 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 19:24:38.102441 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094340 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 19:24:38.102441 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094343 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 19:24:38.102441 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094345 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 19:24:38.102441 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094348 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 19:24:38.102441 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094351 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 19:24:38.102441 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094355 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 19:24:38.102441 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094357 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 19:24:38.102441 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094362 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 19:24:38.102441 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094364 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 19:24:38.102441 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094367 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 19:24:38.102905 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094370 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 19:24:38.102905 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094373 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 19:24:38.102905 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094376 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 19:24:38.102905 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094391 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 19:24:38.102905 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094394 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 19:24:38.102905 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094396 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 19:24:38.102905 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094399 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 19:24:38.102905 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094401 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 19:24:38.102905 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094404 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 19:24:38.102905 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094406 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 19:24:38.102905 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094409 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 19:24:38.102905 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094411 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 19:24:38.102905 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094414 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 19:24:38.102905 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094417 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 19:24:38.102905 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094419 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 19:24:38.102905 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094422 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 19:24:38.102905 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094424 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 19:24:38.102905 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094427 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 19:24:38.102905 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094429 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 19:24:38.102905 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094432 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 19:24:38.103367 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094435 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 19:24:38.103367 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094438 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 19:24:38.103367 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094440 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 19:24:38.103367 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.094443 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 19:24:38.103367 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.095021 2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 30 19:24:38.103367 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.102082 2556 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 30 19:24:38.103367 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.102095 2556 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 30 19:24:38.103367 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.102936 2556 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 30 19:24:38.103367 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.102948 2556 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 30 19:24:38.103367 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103001 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 19:24:38.103367 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103006 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 19:24:38.103367 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103009 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 19:24:38.103367 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103013 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 19:24:38.103367 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103016 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 19:24:38.103367 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103019 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 19:24:38.103749 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103021 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 19:24:38.103749 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103024 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 19:24:38.103749 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103028 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 19:24:38.103749 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103032 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 19:24:38.103749 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103035 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 19:24:38.103749 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103038 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 19:24:38.103749 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103042 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 19:24:38.103749 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103046 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 19:24:38.103749 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103048 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 19:24:38.103749 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103051 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 19:24:38.103749 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103054 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 19:24:38.103749 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103058 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 19:24:38.103749 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103061 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 19:24:38.103749 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103063 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 19:24:38.103749 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103066 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 19:24:38.103749 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103069 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 19:24:38.103749 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103072 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 19:24:38.103749 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103074 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 19:24:38.103749 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103077 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 19:24:38.104197 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103079 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 19:24:38.104197 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103082 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 19:24:38.104197 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103085 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 19:24:38.104197 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103088 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 19:24:38.104197 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103092 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 19:24:38.104197 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103094 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 19:24:38.104197 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103097 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 19:24:38.104197 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103100 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 19:24:38.104197 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103102 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 19:24:38.104197 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103105 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 19:24:38.104197 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103107 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 19:24:38.104197 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103110 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 19:24:38.104197 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103112 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 19:24:38.104197 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103115 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 19:24:38.104197 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103117 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 19:24:38.104197 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103120 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 19:24:38.104197 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103122 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 19:24:38.104197 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103125 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 19:24:38.104197 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103127 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 19:24:38.104197 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103130 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 19:24:38.104660 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103132 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 19:24:38.104660 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103135 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 19:24:38.104660 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103137 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 19:24:38.104660 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103140 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 19:24:38.104660 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103142 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 19:24:38.104660 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103145 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 19:24:38.104660 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103147 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 19:24:38.104660 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103149 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 19:24:38.104660 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103152 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 19:24:38.104660 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103154 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 19:24:38.104660 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103157 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 19:24:38.104660 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103159 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 19:24:38.104660 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103162 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 19:24:38.104660 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103164 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 19:24:38.104660 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103166 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 19:24:38.104660 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103169 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 19:24:38.104660 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103172 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 19:24:38.104660 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103175 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 19:24:38.104660 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103178 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 19:24:38.105102 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103180 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 19:24:38.105102 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103183 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 19:24:38.105102 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103185 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 19:24:38.105102 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103188 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 19:24:38.105102 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103190 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 19:24:38.105102 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103192 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 19:24:38.105102 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103195 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 19:24:38.105102 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103198 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 19:24:38.105102 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103200 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 19:24:38.105102 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103203 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 19:24:38.105102 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103205 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 19:24:38.105102 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103208 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 19:24:38.105102 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103210 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 19:24:38.105102 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103213 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 19:24:38.105102 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103215 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 30 19:24:38.105102 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103218 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 19:24:38.105102 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103220 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 19:24:38.105102 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103223 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 19:24:38.105102 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103225 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 19:24:38.105102 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103228 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 19:24:38.105644 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103230 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 19:24:38.105644 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103233 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 19:24:38.105644 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103235 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 19:24:38.105644 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103238 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 19:24:38.105644 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103240 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 19:24:38.105644 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103243 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 19:24:38.105644 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103245 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 19:24:38.105644 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103247 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 19:24:38.105644 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103251 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 19:24:38.105644 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103254 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 19:24:38.105644 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103257 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 19:24:38.105644 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103260 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 19:24:38.105644 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103262 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 19:24:38.105644 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103265 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 19:24:38.105644 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103267 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 19:24:38.105997 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.103272 2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 30 19:24:38.105997 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103372 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 19:24:38.105997 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103377 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 19:24:38.105997 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103396 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 19:24:38.105997 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103399 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 19:24:38.105997 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103401 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 19:24:38.105997 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103404 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 19:24:38.105997 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103407 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 19:24:38.105997 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103409 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 19:24:38.105997 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103412 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 19:24:38.105997 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103415 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 19:24:38.105997 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103417 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 19:24:38.105997 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103420 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 19:24:38.105997 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103423 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 19:24:38.105997 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103425 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 19:24:38.106363 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103428 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 19:24:38.106363 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103430 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 19:24:38.106363 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103434 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 19:24:38.106363 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103436 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 19:24:38.106363 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103438 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 19:24:38.106363 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103441 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 19:24:38.106363 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103443 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 19:24:38.106363 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103447 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 19:24:38.106363 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103450 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 19:24:38.106363 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103453 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 19:24:38.106363 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103456 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 19:24:38.106363 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103459 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 19:24:38.106363 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103462 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 19:24:38.106363 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103464 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 19:24:38.106363 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103468 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 19:24:38.106363 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103470 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 19:24:38.106363 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103473 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 19:24:38.106363 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103491 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 19:24:38.106363 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103496 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 19:24:38.106825 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103499 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 19:24:38.106825 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103502 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 19:24:38.106825 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103505 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 19:24:38.106825 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103508 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 19:24:38.106825 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103511 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 19:24:38.106825 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103514 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 19:24:38.106825 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103516 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 19:24:38.106825 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103519 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 19:24:38.106825 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103522 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 19:24:38.106825 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103525 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 19:24:38.106825 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103527 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 19:24:38.106825 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103529 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 19:24:38.106825 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103532 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 19:24:38.106825 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103534 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 19:24:38.106825 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103536 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 19:24:38.106825 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103539 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 19:24:38.106825 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103541 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 19:24:38.106825 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103544 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 19:24:38.106825 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103546 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 19:24:38.107277 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103548 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 19:24:38.107277 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103551 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 19:24:38.107277 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103553 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 19:24:38.107277 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103555 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 19:24:38.107277 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103558 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 19:24:38.107277 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103561 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 19:24:38.107277 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103564 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 19:24:38.107277 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103566 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 19:24:38.107277 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103569 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 19:24:38.107277 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103572 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 19:24:38.107277 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103574 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 19:24:38.107277 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103577 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 19:24:38.107277 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103579 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 19:24:38.107277 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103581 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 19:24:38.107277 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103584 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 19:24:38.107277 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103588 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 19:24:38.107277 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103591 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 19:24:38.107277 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103593 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 19:24:38.107277 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103596 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 19:24:38.107734 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103599 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 19:24:38.107734 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103602 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 19:24:38.107734 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103604 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 19:24:38.107734 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103607 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 30 19:24:38.107734 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103609 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 19:24:38.107734 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103611 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 19:24:38.107734 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103614 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 19:24:38.107734 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103617 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 19:24:38.107734 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103619 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 19:24:38.107734 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103621 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 19:24:38.107734 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103624 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 19:24:38.107734 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103626 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 19:24:38.107734 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103629 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 19:24:38.107734 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103631 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 19:24:38.107734 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103633 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 19:24:38.107734 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103636 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 19:24:38.107734 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103638 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 19:24:38.107734 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103641 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 19:24:38.107734 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103643 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 19:24:38.107734 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103646 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 19:24:38.108201 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103648 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 19:24:38.108201 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103650 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 19:24:38.108201 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103653 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 19:24:38.108201 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103656 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 19:24:38.108201 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103658 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 19:24:38.108201 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103660 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 19:24:38.108201 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103663 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 19:24:38.108201 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:38.103665 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 19:24:38.108201 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.103669 2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 30 19:24:38.108201 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.103760 2556 server.go:962] "Client rotation is on, will bootstrap in background" Jul 30 19:24:38.108201 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.105731 2556 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 30 19:24:38.108201 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.107038 2556 server.go:1019] "Starting client certificate rotation" Jul 30 19:24:38.108201 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.107136 2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 30 19:24:38.108201 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.107164 2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 30 19:24:38.109950 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.109937 2556 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 30 19:24:38.111180 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.111166 2556 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 30 19:24:38.119860 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.119842 2556 log.go:25] "Validated CRI v1 runtime API" Jul 30 19:24:38.122203 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.122186 2556 log.go:25] "Validated CRI v1 image API" Jul 30 19:24:38.122951 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.122936 2556 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 30 19:24:38.125161 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.125144 2556 fs.go:135] Filesystem UUIDs: map[310d8f7c-20fb-48d7-ba53-a8f083dca8c7:/dev/nvme0n1p4 6b414d84-b396-464c-ab80-59e0f0bf6415:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 30 19:24:38.125230 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.125160 2556 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 30 19:24:38.129906 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.129801 2556 manager.go:216] Machine: {Timestamp:2026-07-30 19:24:38.128534546 +0000 UTC m=+0.348093449 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3200761 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23bc2a553189ac9264540df0ac4efc SystemUUID:ec23bc2a-5531-89ac-9264-540df0ac4efc BootID:5de45d0f-52e7-4a67-96e8-7756b5c5dd42 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:55:97:a8:61:8d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:55:97:a8:61:8d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:76:a0:30:ff:4e:f1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 30 19:24:38.129906 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.129905 2556 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 30 19:24:38.130011 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.129969 2556 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 30 19:24:38.130946 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.130917 2556 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 30 19:24:38.131062 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.130949 2556 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-105.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 30 19:24:38.131739 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.131729 2556 topology_manager.go:138] "Creating topology manager with none policy" Jul 30 19:24:38.131774 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.131741 2556 container_manager_linux.go:309] "Creating device plugin manager" Jul 30 19:24:38.131774 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.131748 2556 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 30 19:24:38.132532 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.132522 2556 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 30 19:24:38.133697 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.133688 2556 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 30 19:24:38.136443 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.136428 2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 30 19:24:38.137090 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.137079 2556 state_mem.go:36] "Initialized new in-memory state store" Jul 30 19:24:38.137190 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.137182 2556 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 30 19:24:38.137219 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.137209 2556 kubelet.go:486] "Attempting to sync node with API server" Jul 30 19:24:38.137219 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.137217 2556 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 30 19:24:38.137270 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.137233 2556 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 30 19:24:38.137270 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.137241 2556 kubelet.go:398] "Adding apiserver pod source" Jul 30 19:24:38.137270 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.137249 2556 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 30 19:24:38.137569 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.137558 2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 30 19:24:38.137619 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.137576 2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 30 19:24:38.139711 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.139681 2556 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 30 19:24:38.140927 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.140915 2556 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 30 19:24:38.140965 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.140943 2556 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 30 19:24:38.141486 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.141476 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 30 19:24:38.141525 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.141491 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 30 19:24:38.141525 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.141498 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 30 19:24:38.141525 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.141504 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 30 19:24:38.141525 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.141511 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 30 19:24:38.141525 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.141517 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 30 19:24:38.141695 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.141682 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 30 19:24:38.141723 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.141700 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 30 19:24:38.141723 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.141713 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 30 19:24:38.141776 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.141725 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 30 19:24:38.141776 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.141757 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 30 19:24:38.141776 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.141773 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 30 19:24:38.142336 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.142326 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 30 19:24:38.142365 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.142337 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 30 19:24:38.145510 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.145499 2556 server.go:1268] "Started kubelet" Jul 30 19:24:38.145605 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.145566 2556 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 30 19:24:38.145701 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.145654 2556 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 30 19:24:38.145747 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.145719 2556 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 30 19:24:38.146226 ip-10-0-129-105 systemd[1]: Started Kubernetes Kubelet. Jul 30 19:24:38.146916 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.146843 2556 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 30 19:24:38.150223 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:38.146878 2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 30 19:24:38.150620 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:38.150594 2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-105.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 30 19:24:38.150705 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.150636 2556 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-105.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 30 19:24:38.151890 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.151873 2556 server.go:310] "Adding debug handlers to kubelet server" Jul 30 19:24:38.152216 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:38.152179 2556 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 30 19:24:38.155103 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.155085 2556 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 30 19:24:38.155194 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.155103 2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 30 19:24:38.155295 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:38.155185 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-105.ec2.internal\" not found" Jul 30 19:24:38.155544 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.155512 2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dgrgt" Jul 30 19:24:38.155643 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.155626 2556 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 30 19:24:38.155729 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.155640 2556 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 30 19:24:38.155729 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.155729 2556 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 30 19:24:38.155869 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.155841 2556 reconstruct.go:97] "Volume reconstruction finished" Jul 30 19:24:38.155869 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.155848 2556 reconciler.go:29] "Reconciler: start to sync state" Jul 30 19:24:38.155985 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.155975 2556 factory.go:55] Registering systemd factory Jul 30 19:24:38.156046 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.155989 2556 factory.go:223] Registration of the systemd container factory successfully Jul 30 19:24:38.156204 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.156188 2556 factory.go:153] Registering CRI-O factory Jul 30 19:24:38.156204 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.156203 2556 factory.go:223] Registration of the crio container factory successfully Jul 30 19:24:38.156320 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.156247 2556 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 30 19:24:38.156320 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.156269 2556 factory.go:103] Registering Raw factory Jul 30 19:24:38.156320 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.156284 2556 manager.go:1196] Started watching for new ooms in manager Jul 30 19:24:38.156708 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.156693 2556 manager.go:349] Starting recovery of all containers Jul 30 19:24:38.157025 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:38.155797 2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-105.ec2.internal.18c727bee6d1d61d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-105.ec2.internal,UID:ip-10-0-129-105.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-105.ec2.internal,},FirstTimestamp:2026-07-30 19:24:38.145472029 +0000 UTC m=+0.365030932,LastTimestamp:2026-07-30 19:24:38.145472029 +0000 UTC m=+0.365030932,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-105.ec2.internal,}" Jul 30 19:24:38.160685 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.160524 2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dgrgt" Jul 30 19:24:38.161233 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:38.161214 2556 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 30 19:24:38.161339 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:38.161309 2556 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-105.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 30 19:24:38.165278 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.165263 2556 manager.go:354] Recovery completed Jul 30 19:24:38.169166 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.169153 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 19:24:38.170881 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.170857 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-105.ec2.internal" event="NodeHasSufficientMemory" Jul 30 19:24:38.170881 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.170880 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-105.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 19:24:38.170994 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.170889 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-105.ec2.internal" event="NodeHasSufficientPID" Jul 30 19:24:38.171352 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.171338 2556 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 30 19:24:38.171352 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.171351 2556 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 30 19:24:38.171352 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.171353 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 19:24:38.171522 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.171364 2556 state_mem.go:36] "Initialized new in-memory state store" Jul 30 19:24:38.172035 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.172019 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-105.ec2.internal" event="NodeHasSufficientMemory" Jul 30 19:24:38.172096 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.172049 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-105.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 19:24:38.172096 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.172063 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-105.ec2.internal" event="NodeHasSufficientPID" Jul 30 19:24:38.172504 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.172492 2556 policy_none.go:49] "None policy: Start" Jul 30 19:24:38.172575 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.172508 2556 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 30 19:24:38.172575 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.172521 2556 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 30 19:24:38.173091 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:38.173034 2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-105.ec2.internal.18c727bee8555cf1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-105.ec2.internal,UID:ip-10-0-129-105.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-105.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-105.ec2.internal,},FirstTimestamp:2026-07-30 19:24:38.170868977 +0000 UTC m=+0.390427880,LastTimestamp:2026-07-30 19:24:38.170868977 +0000 UTC m=+0.390427880,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-105.ec2.internal,}" Jul 30 19:24:38.173507 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.173494 2556 policy_none.go:47] "Start" Jul 30 19:24:38.215074 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.214944 2556 manager.go:337] "Starting Device Plugin manager" Jul 30 19:24:38.215167 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:38.215097 2556 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 30 19:24:38.215167 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.215112 2556 server.go:87] "Starting device plugin registration server" Jul 30 19:24:38.215309 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.215298 2556 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 30 19:24:38.215363 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.215310 2556 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 30 19:24:38.215473 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.215456 2556 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 30 19:24:38.215530 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.215525 2556 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 30 19:24:38.215571 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.215533 2556 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 30 19:24:38.215987 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:38.215972 2556 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 30 19:24:38.216061 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:38.216012 2556 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-105.ec2.internal\" not found" Jul 30 19:24:38.315883 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.315846 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 19:24:38.316192 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.316172 2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 30 19:24:38.316603 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.316583 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-105.ec2.internal" event="NodeHasSufficientMemory" Jul 30 19:24:38.316706 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.316620 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-105.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 19:24:38.316706 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.316634 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-105.ec2.internal" event="NodeHasSufficientPID" Jul 30 19:24:38.316706 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.316658 2556 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-105.ec2.internal" Jul 30 19:24:38.317478 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.317450 2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 30 19:24:38.317562 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.317486 2556 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 30 19:24:38.317562 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.317504 2556 kubelet.go:2448] "Starting kubelet main sync loop" Jul 30 19:24:38.317562 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:38.317542 2556 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 30 19:24:38.319588 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.319564 2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 19:24:38.322529 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.322514 2556 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-105.ec2.internal" Jul 30 19:24:38.322593 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:38.322533 2556 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-105.ec2.internal\": node \"ip-10-0-129-105.ec2.internal\" not found" Jul 30 19:24:38.370232 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:38.370185 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-105.ec2.internal\" not found" Jul 30 19:24:38.420671 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.420652 2556 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-105.ec2.internal"] Jul 30 19:24:38.420721 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.420706 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 19:24:38.421476 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.421461 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-105.ec2.internal" event="NodeHasSufficientMemory" Jul 30 19:24:38.421567 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.421492 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-105.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 19:24:38.421567 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.421507 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-105.ec2.internal" event="NodeHasSufficientPID" Jul 30 19:24:38.422712 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.422698 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 19:24:38.422838 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.422823 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal" Jul 30 19:24:38.422882 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.422849 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 19:24:38.423375 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.423355 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-105.ec2.internal" event="NodeHasSufficientMemory" Jul 30 19:24:38.423375 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.423367 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-105.ec2.internal" event="NodeHasSufficientMemory" Jul 30 19:24:38.423375 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.423381 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-105.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 19:24:38.423375 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.423391 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-105.ec2.internal" event="NodeHasSufficientPID" Jul 30 19:24:38.423590 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.423407 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-105.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 19:24:38.423590 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.423421 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-105.ec2.internal" event="NodeHasSufficientPID" Jul 30 19:24:38.424686 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.424673 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-105.ec2.internal" Jul 30 19:24:38.424742 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.424712 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 19:24:38.425450 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.425435 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-105.ec2.internal" event="NodeHasSufficientMemory" Jul 30 19:24:38.425519 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.425473 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-105.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 19:24:38.425519 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.425491 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-105.ec2.internal" event="NodeHasSufficientPID" Jul 30 19:24:38.440969 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:38.440953 2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-105.ec2.internal\" not found" node="ip-10-0-129-105.ec2.internal" Jul 30 19:24:38.445221 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:38.445207 2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-105.ec2.internal\" not found" node="ip-10-0-129-105.ec2.internal" Jul 30 19:24:38.457975 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.457946 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1728156c8efd1097c5a710ba4d71fa3e-config\") pod \"kube-apiserver-proxy-ip-10-0-129-105.ec2.internal\" (UID: \"1728156c8efd1097c5a710ba4d71fa3e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-105.ec2.internal" Jul 30 19:24:38.458030 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.457981 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2af0cc64bc27103c007e4f088d273813-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal\" (UID: \"2af0cc64bc27103c007e4f088d273813\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal" Jul 30 19:24:38.458030 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.458002 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2af0cc64bc27103c007e4f088d273813-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal\" (UID: \"2af0cc64bc27103c007e4f088d273813\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal" Jul 30 19:24:38.471230 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:38.471212 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-105.ec2.internal\" not found" Jul 30 19:24:38.558106 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.558085 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2af0cc64bc27103c007e4f088d273813-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal\" (UID: \"2af0cc64bc27103c007e4f088d273813\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal" Jul 30 19:24:38.558155 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.558111 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2af0cc64bc27103c007e4f088d273813-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal\" (UID: \"2af0cc64bc27103c007e4f088d273813\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal" Jul 30 19:24:38.558155 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.558127 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1728156c8efd1097c5a710ba4d71fa3e-config\") pod \"kube-apiserver-proxy-ip-10-0-129-105.ec2.internal\" (UID: \"1728156c8efd1097c5a710ba4d71fa3e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-105.ec2.internal" Jul 30 19:24:38.558220 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.558168 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1728156c8efd1097c5a710ba4d71fa3e-config\") pod \"kube-apiserver-proxy-ip-10-0-129-105.ec2.internal\" (UID: \"1728156c8efd1097c5a710ba4d71fa3e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-105.ec2.internal" Jul 30 19:24:38.558220 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.558199 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2af0cc64bc27103c007e4f088d273813-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal\" (UID: \"2af0cc64bc27103c007e4f088d273813\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal" Jul 30 19:24:38.558281 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.558184 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2af0cc64bc27103c007e4f088d273813-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal\" (UID: \"2af0cc64bc27103c007e4f088d273813\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal" Jul 30 19:24:38.571645 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:38.571617 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-105.ec2.internal\" not found" Jul 30 19:24:38.671857 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:38.671781 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-105.ec2.internal\" not found" Jul 30 19:24:38.747813 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.747784 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal" Jul 30 19:24:38.747915 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:38.747787 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-105.ec2.internal" Jul 30 19:24:38.771896 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:38.771869 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-105.ec2.internal\" not found" Jul 30 19:24:38.872919 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:38.872881 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-105.ec2.internal\" not found" Jul 30 19:24:38.973896 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:38.973843 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-105.ec2.internal\" not found" Jul 30 19:24:39.074039 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:39.074012 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-105.ec2.internal\" not found" Jul 30 19:24:39.106722 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:39.106699 2556 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 30 19:24:39.107082 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:39.106823 2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 30 19:24:39.169994 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:39.169964 2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-29 19:19:38 +0000 UTC" deadline="2028-02-21 14:42:53.357396235 +0000 UTC" Jul 30 19:24:39.169994 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:39.169988 2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13699h18m14.187411507s" Jul 30 19:24:39.174737 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:39.174721 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-105.ec2.internal\" not found" Jul 30 19:24:39.211994 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:39.211972 2556 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 19:24:39.274908 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:39.274851 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-105.ec2.internal\" not found" Jul 30 19:24:39.297784 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:39.297762 2556 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 19:24:39.319093 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:39.319060 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1728156c8efd1097c5a710ba4d71fa3e.slice/crio-902d51ac60f0fe97bc11e15cc839559e32c91430751b6f7b7b5dcbe0cf55b507 WatchSource:0}: Error finding container 902d51ac60f0fe97bc11e15cc839559e32c91430751b6f7b7b5dcbe0cf55b507: Status 404 returned error can't find the container with id 902d51ac60f0fe97bc11e15cc839559e32c91430751b6f7b7b5dcbe0cf55b507 Jul 30 19:24:39.320216 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:39.320195 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2af0cc64bc27103c007e4f088d273813.slice/crio-d6a06f1a14df16ac2b87776ea281b8fe40ebe1f5c420f9ee42422c62efd38cf2 WatchSource:0}: Error finding container d6a06f1a14df16ac2b87776ea281b8fe40ebe1f5c420f9ee42422c62efd38cf2: Status 404 returned error can't find the container with id d6a06f1a14df16ac2b87776ea281b8fe40ebe1f5c420f9ee42422c62efd38cf2 Jul 30 19:24:39.325430 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:39.325415 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 30 19:24:39.356197 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:39.356177 2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal" Jul 30 19:24:39.370502 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:39.370487 2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 30 19:24:39.371580 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:39.371568 2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-105.ec2.internal" Jul 30 19:24:39.380469 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:39.380444 2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 30 19:24:39.436665 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:39.436648 2556 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 19:24:39.967432 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:39.967407 2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 19:24:40.138106 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.138046 2556 apiserver.go:52] "Watching apiserver" Jul 30 19:24:40.151876 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.151846 2556 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 30 19:24:40.153241 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.153214 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-6tpv8","openshift-cluster-node-tuning-operator/tuned-qjcpf","openshift-image-registry/node-ca-rfjmx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal","openshift-multus/multus-additional-cni-plugins-r8fml","openshift-multus/multus-ddrgc","openshift-multus/network-metrics-daemon-vxwhr","openshift-network-diagnostics/network-check-target-czm5g","kube-system/konnectivity-agent-ttglk","kube-system/kube-apiserver-proxy-ip-10-0-129-105.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lp98t","openshift-dns/node-resolver-5p5fm","openshift-network-operator/iptables-alerter-dhdl2","openshift-ovn-kubernetes/ovnkube-node-795qw"] Jul 30 19:24:40.154990 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.154963 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5p5fm" Jul 30 19:24:40.156280 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.155995 2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 30 19:24:40.157020 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.156887 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qjcpf" Jul 30 19:24:40.157483 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.157320 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 30 19:24:40.157573 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.157484 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 30 19:24:40.157622 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.157574 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xmj44\"" Jul 30 19:24:40.158892 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.158869 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rfjmx" Jul 30 19:24:40.159573 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.159556 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dzj4w\"" Jul 30 19:24:40.160016 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.159737 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 30 19:24:40.160016 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.159851 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 30 19:24:40.160205 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.160173 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.161185 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.161144 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 30 19:24:40.163215 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.161604 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-r8fml" Jul 30 19:24:40.163215 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.161857 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rkn2m\"" Jul 30 19:24:40.163215 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.161978 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 30 19:24:40.163215 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.162599 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 30 19:24:40.163215 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.162728 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 30 19:24:40.163215 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.163005 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 30 19:24:40.163215 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.163040 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 30 19:24:40.163697 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.163513 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 30 19:24:40.165354 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.164148 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lj4fp\"" Jul 30 19:24:40.165354 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.164386 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 30 19:24:40.165354 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.164702 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 30 19:24:40.165354 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.164917 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nrmjb\"" Jul 30 19:24:40.165899 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.165429 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 30 19:24:40.165899 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.165771 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 30 19:24:40.166034 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.165973 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 30 19:24:40.166795 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.166161 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ddrgc" Jul 30 19:24:40.166795 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.166311 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 30 19:24:40.166943 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.166858 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 30 19:24:40.170130 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.167301 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxwhr" Jul 30 19:24:40.170130 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:40.167432 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxwhr" podUID="9547ff50-df97-486d-85eb-c253420d0e73" Jul 30 19:24:40.170130 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.167778 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s97k\" (UniqueName: \"kubernetes.io/projected/e9a5febb-adab-42b2-a740-a057ada6294a-kube-api-access-6s97k\") pod \"node-resolver-5p5fm\" (UID: \"e9a5febb-adab-42b2-a740-a057ada6294a\") " pod="openshift-dns/node-resolver-5p5fm" Jul 30 19:24:40.170130 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.167807 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/87b39982-2538-4dcf-baa2-6c522b0c4896-host\") pod \"tuned-qjcpf\" (UID: \"87b39982-2538-4dcf-baa2-6c522b0c4896\") " pod="openshift-cluster-node-tuning-operator/tuned-qjcpf" Jul 30 19:24:40.170130 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.167832 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/87b39982-2538-4dcf-baa2-6c522b0c4896-etc-sysconfig\") pod \"tuned-qjcpf\" (UID: \"87b39982-2538-4dcf-baa2-6c522b0c4896\") " pod="openshift-cluster-node-tuning-operator/tuned-qjcpf" Jul 30 19:24:40.170130 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.167855 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/87b39982-2538-4dcf-baa2-6c522b0c4896-etc-sysctl-d\") pod \"tuned-qjcpf\" (UID: \"87b39982-2538-4dcf-baa2-6c522b0c4896\") " pod="openshift-cluster-node-tuning-operator/tuned-qjcpf" Jul 30 19:24:40.170130 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.167876 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/87b39982-2538-4dcf-baa2-6c522b0c4896-etc-systemd\") pod \"tuned-qjcpf\" (UID: \"87b39982-2538-4dcf-baa2-6c522b0c4896\") " pod="openshift-cluster-node-tuning-operator/tuned-qjcpf" Jul 30 19:24:40.170130 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.167899 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/87b39982-2538-4dcf-baa2-6c522b0c4896-run\") pod \"tuned-qjcpf\" (UID: \"87b39982-2538-4dcf-baa2-6c522b0c4896\") " pod="openshift-cluster-node-tuning-operator/tuned-qjcpf" Jul 30 19:24:40.170130 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.167926 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/87b39982-2538-4dcf-baa2-6c522b0c4896-sys\") pod \"tuned-qjcpf\" (UID: \"87b39982-2538-4dcf-baa2-6c522b0c4896\") " pod="openshift-cluster-node-tuning-operator/tuned-qjcpf" Jul 30 19:24:40.170130 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.167954 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e9a5febb-adab-42b2-a740-a057ada6294a-hosts-file\") pod \"node-resolver-5p5fm\" (UID: \"e9a5febb-adab-42b2-a740-a057ada6294a\") " pod="openshift-dns/node-resolver-5p5fm" Jul 30 19:24:40.170130 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.167978 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9a5febb-adab-42b2-a740-a057ada6294a-tmp-dir\") pod \"node-resolver-5p5fm\" (UID: \"e9a5febb-adab-42b2-a740-a057ada6294a\") " pod="openshift-dns/node-resolver-5p5fm" Jul 30 19:24:40.170130 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.168021 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87b39982-2538-4dcf-baa2-6c522b0c4896-var-lib-kubelet\") pod \"tuned-qjcpf\" (UID: \"87b39982-2538-4dcf-baa2-6c522b0c4896\") " pod="openshift-cluster-node-tuning-operator/tuned-qjcpf" Jul 30 19:24:40.170130 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.168046 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/87b39982-2538-4dcf-baa2-6c522b0c4896-etc-tuned\") pod \"tuned-qjcpf\" (UID: \"87b39982-2538-4dcf-baa2-6c522b0c4896\") " pod="openshift-cluster-node-tuning-operator/tuned-qjcpf" Jul 30 19:24:40.170130 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.168110 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/87b39982-2538-4dcf-baa2-6c522b0c4896-etc-modprobe-d\") pod \"tuned-qjcpf\" (UID: \"87b39982-2538-4dcf-baa2-6c522b0c4896\") " pod="openshift-cluster-node-tuning-operator/tuned-qjcpf" Jul 30 19:24:40.170130 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.168213 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87b39982-2538-4dcf-baa2-6c522b0c4896-etc-kubernetes\") pod \"tuned-qjcpf\" (UID: \"87b39982-2538-4dcf-baa2-6c522b0c4896\") " pod="openshift-cluster-node-tuning-operator/tuned-qjcpf" Jul 30 19:24:40.170130 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.168250 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/87b39982-2538-4dcf-baa2-6c522b0c4896-etc-sysctl-conf\") pod \"tuned-qjcpf\" (UID: \"87b39982-2538-4dcf-baa2-6c522b0c4896\") " pod="openshift-cluster-node-tuning-operator/tuned-qjcpf" Jul 30 19:24:40.170130 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.168272 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/87b39982-2538-4dcf-baa2-6c522b0c4896-lib-modules\") pod \"tuned-qjcpf\" (UID: \"87b39982-2538-4dcf-baa2-6c522b0c4896\") " pod="openshift-cluster-node-tuning-operator/tuned-qjcpf" Jul 30 19:24:40.171296 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.168315 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87b39982-2538-4dcf-baa2-6c522b0c4896-tmp\") pod \"tuned-qjcpf\" (UID: \"87b39982-2538-4dcf-baa2-6c522b0c4896\") " pod="openshift-cluster-node-tuning-operator/tuned-qjcpf" Jul 30 19:24:40.171296 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.168337 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cszf7\" (UniqueName: \"kubernetes.io/projected/87b39982-2538-4dcf-baa2-6c522b0c4896-kube-api-access-cszf7\") pod \"tuned-qjcpf\" (UID: \"87b39982-2538-4dcf-baa2-6c522b0c4896\") " pod="openshift-cluster-node-tuning-operator/tuned-qjcpf" Jul 30 19:24:40.171296 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.169272 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 30 19:24:40.171296 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.169521 2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 30 19:24:40.171296 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.169727 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-czm5g" Jul 30 19:24:40.171296 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:40.169800 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-czm5g" podUID="c96c7638-b6ce-4a98-a49d-1bf0fb43edf8" Jul 30 19:24:40.171296 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.169923 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-szk5l\"" Jul 30 19:24:40.171296 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.170312 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ttglk" Jul 30 19:24:40.171767 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.171665 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dhdl2" Jul 30 19:24:40.174000 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.173108 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lp98t" Jul 30 19:24:40.174000 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.173563 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 30 19:24:40.174171 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.174157 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-trzh8\"" Jul 30 19:24:40.174680 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.173962 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-v7vb2\"" Jul 30 19:24:40.174680 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.174300 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 30 19:24:40.174680 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.174372 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 30 19:24:40.174680 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.174498 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 30 19:24:40.174680 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.174674 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tpv8" Jul 30 19:24:40.175116 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:40.174730 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6tpv8" podUID="8f32c8f1-7bf6-4b84-8205-555365c55861" Jul 30 19:24:40.175116 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.174964 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 30 19:24:40.179993 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.177217 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-86lv5\"" Jul 30 19:24:40.179993 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.177519 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 30 19:24:40.179993 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.179083 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 30 19:24:40.179993 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.179597 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 30 19:24:40.190835 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.190812 2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rn7xv" Jul 30 19:24:40.198644 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.198618 2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rn7xv" Jul 30 19:24:40.257515 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.257495 2556 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 30 19:24:40.269575 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.269548 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4cdd383f-6493-4d83-80fa-f4ce17e8416e-etc-openvswitch\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.269680 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.269590 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4cdd383f-6493-4d83-80fa-f4ce17e8416e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.269680 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.269621 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/87b39982-2538-4dcf-baa2-6c522b0c4896-etc-systemd\") pod \"tuned-qjcpf\" (UID: \"87b39982-2538-4dcf-baa2-6c522b0c4896\") " pod="openshift-cluster-node-tuning-operator/tuned-qjcpf" Jul 30 19:24:40.269680 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.269647 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/87b39982-2538-4dcf-baa2-6c522b0c4896-sys\") pod \"tuned-qjcpf\" (UID: \"87b39982-2538-4dcf-baa2-6c522b0c4896\") " pod="openshift-cluster-node-tuning-operator/tuned-qjcpf" Jul 30 19:24:40.269680 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.269669 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47812e97-b92a-460e-8383-6728edc33f73-cnibin\") pod \"multus-ddrgc\" (UID: \"47812e97-b92a-460e-8383-6728edc33f73\") " pod="openshift-multus/multus-ddrgc" Jul 30 19:24:40.269884 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.269691 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/47812e97-b92a-460e-8383-6728edc33f73-host-run-k8s-cni-cncf-io\") pod \"multus-ddrgc\" (UID: \"47812e97-b92a-460e-8383-6728edc33f73\") " pod="openshift-multus/multus-ddrgc" Jul 30 19:24:40.269884 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.269717 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/47812e97-b92a-460e-8383-6728edc33f73-host-var-lib-cni-multus\") pod \"multus-ddrgc\" (UID: \"47812e97-b92a-460e-8383-6728edc33f73\") " pod="openshift-multus/multus-ddrgc" Jul 30 19:24:40.269884 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.269736 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/87b39982-2538-4dcf-baa2-6c522b0c4896-etc-systemd\") pod \"tuned-qjcpf\" (UID: \"87b39982-2538-4dcf-baa2-6c522b0c4896\") " pod="openshift-cluster-node-tuning-operator/tuned-qjcpf" Jul 30 19:24:40.269884 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.269742 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fb61e5ec-e983-407c-962c-1552688cbe5c-agent-certs\") pod \"konnectivity-agent-ttglk\" (UID: \"fb61e5ec-e983-407c-962c-1552688cbe5c\") " pod="kube-system/konnectivity-agent-ttglk" Jul 30 19:24:40.269884 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.269807 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/87b39982-2538-4dcf-baa2-6c522b0c4896-sys\") pod \"tuned-qjcpf\" (UID: \"87b39982-2538-4dcf-baa2-6c522b0c4896\") " pod="openshift-cluster-node-tuning-operator/tuned-qjcpf" Jul 30 19:24:40.269884 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.269849 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d1e86eb9-30b6-4e63-8046-3867c587e8f9-device-dir\") pod \"aws-ebs-csi-driver-node-lp98t\" (UID: \"d1e86eb9-30b6-4e63-8046-3867c587e8f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lp98t" Jul 30 19:24:40.269884 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.269878 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4cdd383f-6493-4d83-80fa-f4ce17e8416e-log-socket\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.270169 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.269907 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e9a5febb-adab-42b2-a740-a057ada6294a-hosts-file\") pod \"node-resolver-5p5fm\" (UID: \"e9a5febb-adab-42b2-a740-a057ada6294a\") " pod="openshift-dns/node-resolver-5p5fm" Jul 30 19:24:40.270169 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.269976 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e9a5febb-adab-42b2-a740-a057ada6294a-hosts-file\") pod \"node-resolver-5p5fm\" (UID: \"e9a5febb-adab-42b2-a740-a057ada6294a\") " pod="openshift-dns/node-resolver-5p5fm" Jul 30 19:24:40.270169 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.269977 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fb61e5ec-e983-407c-962c-1552688cbe5c-konnectivity-ca\") pod \"konnectivity-agent-ttglk\" (UID: \"fb61e5ec-e983-407c-962c-1552688cbe5c\") " pod="kube-system/konnectivity-agent-ttglk" Jul 30 19:24:40.270169 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.270003 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4cdd383f-6493-4d83-80fa-f4ce17e8416e-run-systemd\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.270169 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.270023 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4cdd383f-6493-4d83-80fa-f4ce17e8416e-host-run-ovn-kubernetes\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.270169 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.270089 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdx6n\" (UniqueName: \"kubernetes.io/projected/4cdd383f-6493-4d83-80fa-f4ce17e8416e-kube-api-access-xdx6n\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.270169 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.270109 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/87b39982-2538-4dcf-baa2-6c522b0c4896-etc-modprobe-d\") pod \"tuned-qjcpf\" (UID: \"87b39982-2538-4dcf-baa2-6c522b0c4896\") " pod="openshift-cluster-node-tuning-operator/tuned-qjcpf" Jul 30 19:24:40.270169 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.270124 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/87b39982-2538-4dcf-baa2-6c522b0c4896-lib-modules\") pod \"tuned-qjcpf\" (UID: \"87b39982-2538-4dcf-baa2-6c522b0c4896\") " pod="openshift-cluster-node-tuning-operator/tuned-qjcpf" Jul 30 19:24:40.270169 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.270148 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cszf7\" (UniqueName: \"kubernetes.io/projected/87b39982-2538-4dcf-baa2-6c522b0c4896-kube-api-access-cszf7\") pod \"tuned-qjcpf\" (UID: \"87b39982-2538-4dcf-baa2-6c522b0c4896\") " pod="openshift-cluster-node-tuning-operator/tuned-qjcpf" Jul 30 19:24:40.270169 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.270170 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/47812e97-b92a-460e-8383-6728edc33f73-host-run-multus-certs\") pod \"multus-ddrgc\" (UID: \"47812e97-b92a-460e-8383-6728edc33f73\") " pod="openshift-multus/multus-ddrgc" Jul 30 19:24:40.270571 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.270186 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4cdd383f-6493-4d83-80fa-f4ce17e8416e-host-kubelet\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.270571 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.270201 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4cdd383f-6493-4d83-80fa-f4ce17e8416e-node-log\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.270571 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.270225 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4cdd383f-6493-4d83-80fa-f4ce17e8416e-host-cni-bin\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.270571 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.270250 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/87b39982-2538-4dcf-baa2-6c522b0c4896-etc-modprobe-d\") pod \"tuned-qjcpf\" (UID: \"87b39982-2538-4dcf-baa2-6c522b0c4896\") " pod="openshift-cluster-node-tuning-operator/tuned-qjcpf" Jul 30 19:24:40.270571 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.270269 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/87b39982-2538-4dcf-baa2-6c522b0c4896-lib-modules\") pod \"tuned-qjcpf\" (UID: \"87b39982-2538-4dcf-baa2-6c522b0c4896\") " pod="openshift-cluster-node-tuning-operator/tuned-qjcpf" Jul 30 19:24:40.270571 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.270268 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6ed49186-8fa8-4875-8da3-b74f8fc2b712-os-release\") pod \"multus-additional-cni-plugins-r8fml\" (UID: \"6ed49186-8fa8-4875-8da3-b74f8fc2b712\") " pod="openshift-multus/multus-additional-cni-plugins-r8fml" Jul 30 19:24:40.270571 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.270312 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/47812e97-b92a-460e-8383-6728edc33f73-multus-socket-dir-parent\") pod \"multus-ddrgc\" (UID: \"47812e97-b92a-460e-8383-6728edc33f73\") " pod="openshift-multus/multus-ddrgc" Jul 30 19:24:40.270571 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.270349 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9547ff50-df97-486d-85eb-c253420d0e73-metrics-certs\") pod \"network-metrics-daemon-vxwhr\" (UID: \"9547ff50-df97-486d-85eb-c253420d0e73\") " pod="openshift-multus/network-metrics-daemon-vxwhr" Jul 30 19:24:40.270571 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.270375 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl6vj\" (UniqueName: \"kubernetes.io/projected/9547ff50-df97-486d-85eb-c253420d0e73-kube-api-access-xl6vj\") pod \"network-metrics-daemon-vxwhr\" (UID: \"9547ff50-df97-486d-85eb-c253420d0e73\") " pod="openshift-multus/network-metrics-daemon-vxwhr" Jul 30 19:24:40.270571 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.270410 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/87b39982-2538-4dcf-baa2-6c522b0c4896-etc-sysconfig\") pod \"tuned-qjcpf\" (UID: \"87b39982-2538-4dcf-baa2-6c522b0c4896\") " pod="openshift-cluster-node-tuning-operator/tuned-qjcpf" Jul 30 19:24:40.270571 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.270444 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/87b39982-2538-4dcf-baa2-6c522b0c4896-etc-sysctl-d\") pod \"tuned-qjcpf\" (UID: \"87b39982-2538-4dcf-baa2-6c522b0c4896\") " pod="openshift-cluster-node-tuning-operator/tuned-qjcpf" Jul 30 19:24:40.270571 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.270471 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/87b39982-2538-4dcf-baa2-6c522b0c4896-etc-sysconfig\") pod \"tuned-qjcpf\" (UID: \"87b39982-2538-4dcf-baa2-6c522b0c4896\") " pod="openshift-cluster-node-tuning-operator/tuned-qjcpf" Jul 30 19:24:40.270571 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.270472 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/47812e97-b92a-460e-8383-6728edc33f73-host-run-netns\") pod \"multus-ddrgc\" (UID: \"47812e97-b92a-460e-8383-6728edc33f73\") " pod="openshift-multus/multus-ddrgc" Jul 30 19:24:40.270571 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.270519 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/78e98956-00da-43a5-984d-6b90bbc5f2ca-iptables-alerter-script\") pod \"iptables-alerter-dhdl2\" (UID: \"78e98956-00da-43a5-984d-6b90bbc5f2ca\") " pod="openshift-network-operator/iptables-alerter-dhdl2" Jul 30 19:24:40.270571 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.270574 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4cdd383f-6493-4d83-80fa-f4ce17e8416e-ovnkube-script-lib\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.271010 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.270607 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/87b39982-2538-4dcf-baa2-6c522b0c4896-etc-tuned\") pod \"tuned-qjcpf\" (UID: \"87b39982-2538-4dcf-baa2-6c522b0c4896\") " pod="openshift-cluster-node-tuning-operator/tuned-qjcpf" Jul 30 19:24:40.271010 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.270623 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/87b39982-2538-4dcf-baa2-6c522b0c4896-etc-sysctl-d\") pod \"tuned-qjcpf\" (UID: \"87b39982-2538-4dcf-baa2-6c522b0c4896\") " pod="openshift-cluster-node-tuning-operator/tuned-qjcpf" Jul 30 19:24:40.271010 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.270633 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/47812e97-b92a-460e-8383-6728edc33f73-cni-binary-copy\") pod \"multus-ddrgc\" (UID: \"47812e97-b92a-460e-8383-6728edc33f73\") " pod="openshift-multus/multus-ddrgc" Jul 30 19:24:40.271010 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.270662 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxfgl\" (UniqueName: \"kubernetes.io/projected/78e98956-00da-43a5-984d-6b90bbc5f2ca-kube-api-access-cxfgl\") pod \"iptables-alerter-dhdl2\" (UID: \"78e98956-00da-43a5-984d-6b90bbc5f2ca\") " pod="openshift-network-operator/iptables-alerter-dhdl2" Jul 30 19:24:40.271010 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.270698 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d1e86eb9-30b6-4e63-8046-3867c587e8f9-socket-dir\") pod \"aws-ebs-csi-driver-node-lp98t\" (UID: \"d1e86eb9-30b6-4e63-8046-3867c587e8f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lp98t" Jul 30 19:24:40.271010 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.270726 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4cdd383f-6493-4d83-80fa-f4ce17e8416e-var-lib-openvswitch\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.271010 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.270752 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87b39982-2538-4dcf-baa2-6c522b0c4896-etc-kubernetes\") pod \"tuned-qjcpf\" (UID: \"87b39982-2538-4dcf-baa2-6c522b0c4896\") " pod="openshift-cluster-node-tuning-operator/tuned-qjcpf" Jul 30 19:24:40.271010 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.270785 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8f32c8f1-7bf6-4b84-8205-555365c55861-dbus\") pod \"global-pull-secret-syncer-6tpv8\" (UID: \"8f32c8f1-7bf6-4b84-8205-555365c55861\") " pod="kube-system/global-pull-secret-syncer-6tpv8" Jul 30 19:24:40.271010 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.270812 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s97k\" (UniqueName: \"kubernetes.io/projected/e9a5febb-adab-42b2-a740-a057ada6294a-kube-api-access-6s97k\") pod \"node-resolver-5p5fm\" (UID: \"e9a5febb-adab-42b2-a740-a057ada6294a\") " pod="openshift-dns/node-resolver-5p5fm" Jul 30 19:24:40.271010 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.270852 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87b39982-2538-4dcf-baa2-6c522b0c4896-etc-kubernetes\") pod \"tuned-qjcpf\" (UID: \"87b39982-2538-4dcf-baa2-6c522b0c4896\") " pod="openshift-cluster-node-tuning-operator/tuned-qjcpf" Jul 30 19:24:40.271010 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.270861 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4cdd383f-6493-4d83-80fa-f4ce17e8416e-systemd-units\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.271010 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.270904 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4cdd383f-6493-4d83-80fa-f4ce17e8416e-host-slash\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.271010 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.271008 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4cdd383f-6493-4d83-80fa-f4ce17e8416e-ovn-node-metrics-cert\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.271494 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.271032 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4cdd383f-6493-4d83-80fa-f4ce17e8416e-host-cni-netd\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.271494 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.271061 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4cdd383f-6493-4d83-80fa-f4ce17e8416e-env-overrides\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.271494 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.271083 2556 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 30 19:24:40.271494 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.271085 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6ed49186-8fa8-4875-8da3-b74f8fc2b712-cnibin\") pod \"multus-additional-cni-plugins-r8fml\" (UID: \"6ed49186-8fa8-4875-8da3-b74f8fc2b712\") " pod="openshift-multus/multus-additional-cni-plugins-r8fml" Jul 30 19:24:40.271494 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.271161 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6ed49186-8fa8-4875-8da3-b74f8fc2b712-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r8fml\" (UID: \"6ed49186-8fa8-4875-8da3-b74f8fc2b712\") " pod="openshift-multus/multus-additional-cni-plugins-r8fml" Jul 30 19:24:40.271494 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.271195 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr5f9\" (UniqueName: \"kubernetes.io/projected/6ed49186-8fa8-4875-8da3-b74f8fc2b712-kube-api-access-dr5f9\") pod \"multus-additional-cni-plugins-r8fml\" (UID: \"6ed49186-8fa8-4875-8da3-b74f8fc2b712\") " pod="openshift-multus/multus-additional-cni-plugins-r8fml" Jul 30 19:24:40.271494 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.271237 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d1e86eb9-30b6-4e63-8046-3867c587e8f9-etc-selinux\") pod \"aws-ebs-csi-driver-node-lp98t\" (UID: \"d1e86eb9-30b6-4e63-8046-3867c587e8f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lp98t" Jul 30 19:24:40.271494 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.271255 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9a5febb-adab-42b2-a740-a057ada6294a-tmp-dir\") pod \"node-resolver-5p5fm\" (UID: \"e9a5febb-adab-42b2-a740-a057ada6294a\") " pod="openshift-dns/node-resolver-5p5fm" Jul 30 19:24:40.271494 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.271271 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87b39982-2538-4dcf-baa2-6c522b0c4896-var-lib-kubelet\") pod \"tuned-qjcpf\" (UID: \"87b39982-2538-4dcf-baa2-6c522b0c4896\") " pod="openshift-cluster-node-tuning-operator/tuned-qjcpf" Jul 30 19:24:40.271494 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.271294 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47812e97-b92a-460e-8383-6728edc33f73-multus-cni-dir\") pod \"multus-ddrgc\" (UID: \"47812e97-b92a-460e-8383-6728edc33f73\") " pod="openshift-multus/multus-ddrgc" Jul 30 19:24:40.271494 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.271326 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/47812e97-b92a-460e-8383-6728edc33f73-host-var-lib-cni-bin\") pod \"multus-ddrgc\" (UID: \"47812e97-b92a-460e-8383-6728edc33f73\") " pod="openshift-multus/multus-ddrgc" Jul 30 19:24:40.271494 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.271336 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87b39982-2538-4dcf-baa2-6c522b0c4896-var-lib-kubelet\") pod \"tuned-qjcpf\" (UID: \"87b39982-2538-4dcf-baa2-6c522b0c4896\") " pod="openshift-cluster-node-tuning-operator/tuned-qjcpf" Jul 30 19:24:40.271494 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.271355 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47812e97-b92a-460e-8383-6728edc33f73-etc-kubernetes\") pod \"multus-ddrgc\" (UID: \"47812e97-b92a-460e-8383-6728edc33f73\") " pod="openshift-multus/multus-ddrgc" Jul 30 19:24:40.271494 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.271382 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxr62\" (UniqueName: \"kubernetes.io/projected/47812e97-b92a-460e-8383-6728edc33f73-kube-api-access-rxr62\") pod \"multus-ddrgc\" (UID: \"47812e97-b92a-460e-8383-6728edc33f73\") " pod="openshift-multus/multus-ddrgc" Jul 30 19:24:40.271494 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.271422 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d1e86eb9-30b6-4e63-8046-3867c587e8f9-sys-fs\") pod \"aws-ebs-csi-driver-node-lp98t\" (UID: \"d1e86eb9-30b6-4e63-8046-3867c587e8f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lp98t" Jul 30 19:24:40.271494 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.271451 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/87b39982-2538-4dcf-baa2-6c522b0c4896-etc-sysctl-conf\") pod \"tuned-qjcpf\" (UID: \"87b39982-2538-4dcf-baa2-6c522b0c4896\") " pod="openshift-cluster-node-tuning-operator/tuned-qjcpf" Jul 30 19:24:40.271494 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.271488 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87b39982-2538-4dcf-baa2-6c522b0c4896-tmp\") pod \"tuned-qjcpf\" (UID: \"87b39982-2538-4dcf-baa2-6c522b0c4896\") " pod="openshift-cluster-node-tuning-operator/tuned-qjcpf" Jul 30 19:24:40.272339 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.271512 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/47812e97-b92a-460e-8383-6728edc33f73-multus-daemon-config\") pod \"multus-ddrgc\" (UID: \"47812e97-b92a-460e-8383-6728edc33f73\") " pod="openshift-multus/multus-ddrgc" Jul 30 19:24:40.272339 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.271538 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/78e98956-00da-43a5-984d-6b90bbc5f2ca-host-slash\") pod \"iptables-alerter-dhdl2\" (UID: \"78e98956-00da-43a5-984d-6b90bbc5f2ca\") " pod="openshift-network-operator/iptables-alerter-dhdl2" Jul 30 19:24:40.272339 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.271566 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8f32c8f1-7bf6-4b84-8205-555365c55861-original-pull-secret\") pod \"global-pull-secret-syncer-6tpv8\" (UID: \"8f32c8f1-7bf6-4b84-8205-555365c55861\") " pod="kube-system/global-pull-secret-syncer-6tpv8" Jul 30 19:24:40.272339 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.271590 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4cdd383f-6493-4d83-80fa-f4ce17e8416e-run-openvswitch\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.272339 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.271613 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4cdd383f-6493-4d83-80fa-f4ce17e8416e-run-ovn\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.272339 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.271644 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/87b39982-2538-4dcf-baa2-6c522b0c4896-etc-sysctl-conf\") pod \"tuned-qjcpf\" (UID: \"87b39982-2538-4dcf-baa2-6c522b0c4896\") " pod="openshift-cluster-node-tuning-operator/tuned-qjcpf" Jul 30 19:24:40.272339 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.271655 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ed49186-8fa8-4875-8da3-b74f8fc2b712-system-cni-dir\") pod \"multus-additional-cni-plugins-r8fml\" (UID: \"6ed49186-8fa8-4875-8da3-b74f8fc2b712\") " pod="openshift-multus/multus-additional-cni-plugins-r8fml" Jul 30 19:24:40.272339 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.271701 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/87b39982-2538-4dcf-baa2-6c522b0c4896-host\") pod \"tuned-qjcpf\" (UID: \"87b39982-2538-4dcf-baa2-6c522b0c4896\") " pod="openshift-cluster-node-tuning-operator/tuned-qjcpf" Jul 30 19:24:40.272339 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.271761 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/87b39982-2538-4dcf-baa2-6c522b0c4896-host\") pod \"tuned-qjcpf\" (UID: \"87b39982-2538-4dcf-baa2-6c522b0c4896\") " pod="openshift-cluster-node-tuning-operator/tuned-qjcpf" Jul 30 19:24:40.272339 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.271793 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47812e97-b92a-460e-8383-6728edc33f73-system-cni-dir\") pod \"multus-ddrgc\" (UID: \"47812e97-b92a-460e-8383-6728edc33f73\") " pod="openshift-multus/multus-ddrgc" Jul 30 19:24:40.272339 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.271829 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c058e636-437f-4395-9e4a-760cc5fd051c-host\") pod \"node-ca-rfjmx\" (UID: \"c058e636-437f-4395-9e4a-760cc5fd051c\") " pod="openshift-image-registry/node-ca-rfjmx" Jul 30 19:24:40.272339 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.271863 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c058e636-437f-4395-9e4a-760cc5fd051c-serviceca\") pod \"node-ca-rfjmx\" (UID: \"c058e636-437f-4395-9e4a-760cc5fd051c\") " pod="openshift-image-registry/node-ca-rfjmx" Jul 30 19:24:40.272339 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.271908 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6ed49186-8fa8-4875-8da3-b74f8fc2b712-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r8fml\" (UID: \"6ed49186-8fa8-4875-8da3-b74f8fc2b712\") " pod="openshift-multus/multus-additional-cni-plugins-r8fml" Jul 30 19:24:40.272339 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.271935 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6ed49186-8fa8-4875-8da3-b74f8fc2b712-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r8fml\" (UID: \"6ed49186-8fa8-4875-8da3-b74f8fc2b712\") " pod="openshift-multus/multus-additional-cni-plugins-r8fml" Jul 30 19:24:40.272339 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.271964 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/87b39982-2538-4dcf-baa2-6c522b0c4896-run\") pod \"tuned-qjcpf\" (UID: \"87b39982-2538-4dcf-baa2-6c522b0c4896\") " pod="openshift-cluster-node-tuning-operator/tuned-qjcpf" Jul 30 19:24:40.272339 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.271990 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/47812e97-b92a-460e-8383-6728edc33f73-os-release\") pod \"multus-ddrgc\" (UID: \"47812e97-b92a-460e-8383-6728edc33f73\") " pod="openshift-multus/multus-ddrgc" Jul 30 19:24:40.272339 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.272015 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlftb\" (UniqueName: \"kubernetes.io/projected/c058e636-437f-4395-9e4a-760cc5fd051c-kube-api-access-xlftb\") pod \"node-ca-rfjmx\" (UID: \"c058e636-437f-4395-9e4a-760cc5fd051c\") " pod="openshift-image-registry/node-ca-rfjmx" Jul 30 19:24:40.272888 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.272050 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grwl9\" (UniqueName: \"kubernetes.io/projected/d1e86eb9-30b6-4e63-8046-3867c587e8f9-kube-api-access-grwl9\") pod \"aws-ebs-csi-driver-node-lp98t\" (UID: \"d1e86eb9-30b6-4e63-8046-3867c587e8f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lp98t" Jul 30 19:24:40.272888 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.272075 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4cdd383f-6493-4d83-80fa-f4ce17e8416e-host-run-netns\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.272888 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.272112 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4cdd383f-6493-4d83-80fa-f4ce17e8416e-ovnkube-config\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.272888 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.272167 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6ed49186-8fa8-4875-8da3-b74f8fc2b712-cni-binary-copy\") pod \"multus-additional-cni-plugins-r8fml\" (UID: \"6ed49186-8fa8-4875-8da3-b74f8fc2b712\") " pod="openshift-multus/multus-additional-cni-plugins-r8fml" Jul 30 19:24:40.272888 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.272192 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glnkx\" (UniqueName: \"kubernetes.io/projected/c96c7638-b6ce-4a98-a49d-1bf0fb43edf8-kube-api-access-glnkx\") pod \"network-check-target-czm5g\" (UID: \"c96c7638-b6ce-4a98-a49d-1bf0fb43edf8\") " pod="openshift-network-diagnostics/network-check-target-czm5g" Jul 30 19:24:40.272888 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.272217 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d1e86eb9-30b6-4e63-8046-3867c587e8f9-registration-dir\") pod \"aws-ebs-csi-driver-node-lp98t\" (UID: \"d1e86eb9-30b6-4e63-8046-3867c587e8f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lp98t" Jul 30 19:24:40.272888 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.272242 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/47812e97-b92a-460e-8383-6728edc33f73-hostroot\") pod \"multus-ddrgc\" (UID: \"47812e97-b92a-460e-8383-6728edc33f73\") " pod="openshift-multus/multus-ddrgc" Jul 30 19:24:40.272888 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.272257 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/87b39982-2538-4dcf-baa2-6c522b0c4896-run\") pod \"tuned-qjcpf\" (UID: \"87b39982-2538-4dcf-baa2-6c522b0c4896\") " pod="openshift-cluster-node-tuning-operator/tuned-qjcpf" Jul 30 19:24:40.272888 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.272267 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8f32c8f1-7bf6-4b84-8205-555365c55861-kubelet-config\") pod \"global-pull-secret-syncer-6tpv8\" (UID: \"8f32c8f1-7bf6-4b84-8205-555365c55861\") " pod="kube-system/global-pull-secret-syncer-6tpv8" Jul 30 19:24:40.272888 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.272292 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d1e86eb9-30b6-4e63-8046-3867c587e8f9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lp98t\" (UID: \"d1e86eb9-30b6-4e63-8046-3867c587e8f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lp98t" Jul 30 19:24:40.272888 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.272318 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47812e97-b92a-460e-8383-6728edc33f73-host-var-lib-kubelet\") pod \"multus-ddrgc\" (UID: \"47812e97-b92a-460e-8383-6728edc33f73\") " pod="openshift-multus/multus-ddrgc" Jul 30 19:24:40.272888 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.272341 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47812e97-b92a-460e-8383-6728edc33f73-multus-conf-dir\") pod \"multus-ddrgc\" (UID: \"47812e97-b92a-460e-8383-6728edc33f73\") " pod="openshift-multus/multus-ddrgc" Jul 30 19:24:40.272888 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.272363 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9a5febb-adab-42b2-a740-a057ada6294a-tmp-dir\") pod \"node-resolver-5p5fm\" (UID: \"e9a5febb-adab-42b2-a740-a057ada6294a\") " pod="openshift-dns/node-resolver-5p5fm" Jul 30 19:24:40.274537 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.274516 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/87b39982-2538-4dcf-baa2-6c522b0c4896-etc-tuned\") pod \"tuned-qjcpf\" (UID: \"87b39982-2538-4dcf-baa2-6c522b0c4896\") " pod="openshift-cluster-node-tuning-operator/tuned-qjcpf" Jul 30 19:24:40.275185 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.275163 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87b39982-2538-4dcf-baa2-6c522b0c4896-tmp\") pod \"tuned-qjcpf\" (UID: \"87b39982-2538-4dcf-baa2-6c522b0c4896\") " pod="openshift-cluster-node-tuning-operator/tuned-qjcpf" Jul 30 19:24:40.282687 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.282658 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cszf7\" (UniqueName: \"kubernetes.io/projected/87b39982-2538-4dcf-baa2-6c522b0c4896-kube-api-access-cszf7\") pod \"tuned-qjcpf\" (UID: \"87b39982-2538-4dcf-baa2-6c522b0c4896\") " pod="openshift-cluster-node-tuning-operator/tuned-qjcpf" Jul 30 19:24:40.285717 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.285698 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s97k\" (UniqueName: \"kubernetes.io/projected/e9a5febb-adab-42b2-a740-a057ada6294a-kube-api-access-6s97k\") pod \"node-resolver-5p5fm\" (UID: \"e9a5febb-adab-42b2-a740-a057ada6294a\") " pod="openshift-dns/node-resolver-5p5fm" Jul 30 19:24:40.322938 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.322886 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-105.ec2.internal" event={"ID":"1728156c8efd1097c5a710ba4d71fa3e","Type":"ContainerStarted","Data":"902d51ac60f0fe97bc11e15cc839559e32c91430751b6f7b7b5dcbe0cf55b507"} Jul 30 19:24:40.324986 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.324957 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal" event={"ID":"2af0cc64bc27103c007e4f088d273813","Type":"ContainerStarted","Data":"d6a06f1a14df16ac2b87776ea281b8fe40ebe1f5c420f9ee42422c62efd38cf2"} Jul 30 19:24:40.373448 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.373411 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47812e97-b92a-460e-8383-6728edc33f73-host-var-lib-kubelet\") pod \"multus-ddrgc\" (UID: \"47812e97-b92a-460e-8383-6728edc33f73\") " pod="openshift-multus/multus-ddrgc" Jul 30 19:24:40.373613 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.373459 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47812e97-b92a-460e-8383-6728edc33f73-multus-conf-dir\") pod \"multus-ddrgc\" (UID: \"47812e97-b92a-460e-8383-6728edc33f73\") " pod="openshift-multus/multus-ddrgc" Jul 30 19:24:40.373613 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.373485 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4cdd383f-6493-4d83-80fa-f4ce17e8416e-etc-openvswitch\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.373613 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.373513 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4cdd383f-6493-4d83-80fa-f4ce17e8416e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.373613 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.373530 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47812e97-b92a-460e-8383-6728edc33f73-host-var-lib-kubelet\") pod \"multus-ddrgc\" (UID: \"47812e97-b92a-460e-8383-6728edc33f73\") " pod="openshift-multus/multus-ddrgc" Jul 30 19:24:40.373613 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.373542 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47812e97-b92a-460e-8383-6728edc33f73-cnibin\") pod \"multus-ddrgc\" (UID: \"47812e97-b92a-460e-8383-6728edc33f73\") " pod="openshift-multus/multus-ddrgc" Jul 30 19:24:40.373613 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.373540 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47812e97-b92a-460e-8383-6728edc33f73-multus-conf-dir\") pod \"multus-ddrgc\" (UID: \"47812e97-b92a-460e-8383-6728edc33f73\") " pod="openshift-multus/multus-ddrgc" Jul 30 19:24:40.373613 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.373567 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/47812e97-b92a-460e-8383-6728edc33f73-host-run-k8s-cni-cncf-io\") pod \"multus-ddrgc\" (UID: \"47812e97-b92a-460e-8383-6728edc33f73\") " pod="openshift-multus/multus-ddrgc" Jul 30 19:24:40.373613 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.373577 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4cdd383f-6493-4d83-80fa-f4ce17e8416e-etc-openvswitch\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.373613 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.373592 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/47812e97-b92a-460e-8383-6728edc33f73-host-var-lib-cni-multus\") pod \"multus-ddrgc\" (UID: \"47812e97-b92a-460e-8383-6728edc33f73\") " pod="openshift-multus/multus-ddrgc" Jul 30 19:24:40.373613 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.373606 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4cdd383f-6493-4d83-80fa-f4ce17e8416e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.373613 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.373619 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47812e97-b92a-460e-8383-6728edc33f73-cnibin\") pod \"multus-ddrgc\" (UID: \"47812e97-b92a-460e-8383-6728edc33f73\") " pod="openshift-multus/multus-ddrgc" Jul 30 19:24:40.374125 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.373634 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/47812e97-b92a-460e-8383-6728edc33f73-host-var-lib-cni-multus\") pod \"multus-ddrgc\" (UID: \"47812e97-b92a-460e-8383-6728edc33f73\") " pod="openshift-multus/multus-ddrgc" Jul 30 19:24:40.374125 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.373632 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fb61e5ec-e983-407c-962c-1552688cbe5c-agent-certs\") pod \"konnectivity-agent-ttglk\" (UID: \"fb61e5ec-e983-407c-962c-1552688cbe5c\") " pod="kube-system/konnectivity-agent-ttglk" Jul 30 19:24:40.374125 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.373685 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d1e86eb9-30b6-4e63-8046-3867c587e8f9-device-dir\") pod \"aws-ebs-csi-driver-node-lp98t\" (UID: \"d1e86eb9-30b6-4e63-8046-3867c587e8f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lp98t" Jul 30 19:24:40.374125 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.373658 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/47812e97-b92a-460e-8383-6728edc33f73-host-run-k8s-cni-cncf-io\") pod \"multus-ddrgc\" (UID: \"47812e97-b92a-460e-8383-6728edc33f73\") " pod="openshift-multus/multus-ddrgc" Jul 30 19:24:40.374125 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.373710 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4cdd383f-6493-4d83-80fa-f4ce17e8416e-log-socket\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.374125 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.373771 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fb61e5ec-e983-407c-962c-1552688cbe5c-konnectivity-ca\") pod \"konnectivity-agent-ttglk\" (UID: \"fb61e5ec-e983-407c-962c-1552688cbe5c\") " pod="kube-system/konnectivity-agent-ttglk" Jul 30 19:24:40.374125 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.373784 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d1e86eb9-30b6-4e63-8046-3867c587e8f9-device-dir\") pod \"aws-ebs-csi-driver-node-lp98t\" (UID: \"d1e86eb9-30b6-4e63-8046-3867c587e8f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lp98t" Jul 30 19:24:40.374125 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.373801 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4cdd383f-6493-4d83-80fa-f4ce17e8416e-run-systemd\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.374125 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.373825 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4cdd383f-6493-4d83-80fa-f4ce17e8416e-log-socket\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.374125 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.373827 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4cdd383f-6493-4d83-80fa-f4ce17e8416e-host-run-ovn-kubernetes\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.374125 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.373860 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdx6n\" (UniqueName: \"kubernetes.io/projected/4cdd383f-6493-4d83-80fa-f4ce17e8416e-kube-api-access-xdx6n\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.374125 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.373869 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4cdd383f-6493-4d83-80fa-f4ce17e8416e-run-systemd\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.374125 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.373888 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/47812e97-b92a-460e-8383-6728edc33f73-host-run-multus-certs\") pod \"multus-ddrgc\" (UID: \"47812e97-b92a-460e-8383-6728edc33f73\") " pod="openshift-multus/multus-ddrgc" Jul 30 19:24:40.374125 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.373907 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4cdd383f-6493-4d83-80fa-f4ce17e8416e-host-run-ovn-kubernetes\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.374125 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.373934 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/47812e97-b92a-460e-8383-6728edc33f73-host-run-multus-certs\") pod \"multus-ddrgc\" (UID: \"47812e97-b92a-460e-8383-6728edc33f73\") " pod="openshift-multus/multus-ddrgc" Jul 30 19:24:40.374125 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.374007 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4cdd383f-6493-4d83-80fa-f4ce17e8416e-host-kubelet\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.374125 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.374033 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4cdd383f-6493-4d83-80fa-f4ce17e8416e-node-log\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.374988 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.374057 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4cdd383f-6493-4d83-80fa-f4ce17e8416e-host-cni-bin\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.374988 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.374081 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6ed49186-8fa8-4875-8da3-b74f8fc2b712-os-release\") pod \"multus-additional-cni-plugins-r8fml\" (UID: \"6ed49186-8fa8-4875-8da3-b74f8fc2b712\") " pod="openshift-multus/multus-additional-cni-plugins-r8fml" Jul 30 19:24:40.374988 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.374105 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/47812e97-b92a-460e-8383-6728edc33f73-multus-socket-dir-parent\") pod \"multus-ddrgc\" (UID: \"47812e97-b92a-460e-8383-6728edc33f73\") " pod="openshift-multus/multus-ddrgc" Jul 30 19:24:40.374988 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.374125 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4cdd383f-6493-4d83-80fa-f4ce17e8416e-node-log\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.374988 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.374130 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9547ff50-df97-486d-85eb-c253420d0e73-metrics-certs\") pod \"network-metrics-daemon-vxwhr\" (UID: \"9547ff50-df97-486d-85eb-c253420d0e73\") " pod="openshift-multus/network-metrics-daemon-vxwhr" Jul 30 19:24:40.374988 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.374132 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4cdd383f-6493-4d83-80fa-f4ce17e8416e-host-kubelet\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.374988 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.374173 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl6vj\" (UniqueName: \"kubernetes.io/projected/9547ff50-df97-486d-85eb-c253420d0e73-kube-api-access-xl6vj\") pod \"network-metrics-daemon-vxwhr\" (UID: \"9547ff50-df97-486d-85eb-c253420d0e73\") " pod="openshift-multus/network-metrics-daemon-vxwhr" Jul 30 19:24:40.374988 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.374186 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4cdd383f-6493-4d83-80fa-f4ce17e8416e-host-cni-bin\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.374988 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.374203 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/47812e97-b92a-460e-8383-6728edc33f73-host-run-netns\") pod \"multus-ddrgc\" (UID: \"47812e97-b92a-460e-8383-6728edc33f73\") " pod="openshift-multus/multus-ddrgc" Jul 30 19:24:40.374988 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.374219 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6ed49186-8fa8-4875-8da3-b74f8fc2b712-os-release\") pod \"multus-additional-cni-plugins-r8fml\" (UID: \"6ed49186-8fa8-4875-8da3-b74f8fc2b712\") " pod="openshift-multus/multus-additional-cni-plugins-r8fml" Jul 30 19:24:40.374988 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.374231 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/78e98956-00da-43a5-984d-6b90bbc5f2ca-iptables-alerter-script\") pod \"iptables-alerter-dhdl2\" (UID: \"78e98956-00da-43a5-984d-6b90bbc5f2ca\") " pod="openshift-network-operator/iptables-alerter-dhdl2" Jul 30 19:24:40.374988 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.374241 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/47812e97-b92a-460e-8383-6728edc33f73-multus-socket-dir-parent\") pod \"multus-ddrgc\" (UID: \"47812e97-b92a-460e-8383-6728edc33f73\") " pod="openshift-multus/multus-ddrgc" Jul 30 19:24:40.374988 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.374282 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/47812e97-b92a-460e-8383-6728edc33f73-host-run-netns\") pod \"multus-ddrgc\" (UID: \"47812e97-b92a-460e-8383-6728edc33f73\") " pod="openshift-multus/multus-ddrgc" Jul 30 19:24:40.374988 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:40.374303 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 19:24:40.374988 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:40.374391 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9547ff50-df97-486d-85eb-c253420d0e73-metrics-certs podName:9547ff50-df97-486d-85eb-c253420d0e73 nodeName:}" failed. No retries permitted until 2026-07-30 19:24:40.874367685 +0000 UTC m=+3.093926602 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9547ff50-df97-486d-85eb-c253420d0e73-metrics-certs") pod "network-metrics-daemon-vxwhr" (UID: "9547ff50-df97-486d-85eb-c253420d0e73") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 19:24:40.374988 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.374390 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fb61e5ec-e983-407c-962c-1552688cbe5c-konnectivity-ca\") pod \"konnectivity-agent-ttglk\" (UID: \"fb61e5ec-e983-407c-962c-1552688cbe5c\") " pod="kube-system/konnectivity-agent-ttglk" Jul 30 19:24:40.374988 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.374415 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4cdd383f-6493-4d83-80fa-f4ce17e8416e-ovnkube-script-lib\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.375929 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.374447 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/47812e97-b92a-460e-8383-6728edc33f73-cni-binary-copy\") pod \"multus-ddrgc\" (UID: \"47812e97-b92a-460e-8383-6728edc33f73\") " pod="openshift-multus/multus-ddrgc" Jul 30 19:24:40.375929 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.374472 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxfgl\" (UniqueName: \"kubernetes.io/projected/78e98956-00da-43a5-984d-6b90bbc5f2ca-kube-api-access-cxfgl\") pod \"iptables-alerter-dhdl2\" (UID: \"78e98956-00da-43a5-984d-6b90bbc5f2ca\") " pod="openshift-network-operator/iptables-alerter-dhdl2" Jul 30 19:24:40.375929 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.374499 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d1e86eb9-30b6-4e63-8046-3867c587e8f9-socket-dir\") pod \"aws-ebs-csi-driver-node-lp98t\" (UID: \"d1e86eb9-30b6-4e63-8046-3867c587e8f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lp98t" Jul 30 19:24:40.375929 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.374527 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4cdd383f-6493-4d83-80fa-f4ce17e8416e-var-lib-openvswitch\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.375929 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.374556 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8f32c8f1-7bf6-4b84-8205-555365c55861-dbus\") pod \"global-pull-secret-syncer-6tpv8\" (UID: \"8f32c8f1-7bf6-4b84-8205-555365c55861\") " pod="kube-system/global-pull-secret-syncer-6tpv8" Jul 30 19:24:40.375929 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.374590 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4cdd383f-6493-4d83-80fa-f4ce17e8416e-systemd-units\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.375929 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.374615 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4cdd383f-6493-4d83-80fa-f4ce17e8416e-host-slash\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.375929 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.374640 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4cdd383f-6493-4d83-80fa-f4ce17e8416e-ovn-node-metrics-cert\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.375929 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.374664 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4cdd383f-6493-4d83-80fa-f4ce17e8416e-host-cni-netd\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.375929 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.374687 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4cdd383f-6493-4d83-80fa-f4ce17e8416e-env-overrides\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.375929 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.374713 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6ed49186-8fa8-4875-8da3-b74f8fc2b712-cnibin\") pod \"multus-additional-cni-plugins-r8fml\" (UID: \"6ed49186-8fa8-4875-8da3-b74f8fc2b712\") " pod="openshift-multus/multus-additional-cni-plugins-r8fml" Jul 30 19:24:40.375929 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.374734 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d1e86eb9-30b6-4e63-8046-3867c587e8f9-socket-dir\") pod \"aws-ebs-csi-driver-node-lp98t\" (UID: \"d1e86eb9-30b6-4e63-8046-3867c587e8f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lp98t" Jul 30 19:24:40.375929 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.374739 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6ed49186-8fa8-4875-8da3-b74f8fc2b712-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r8fml\" (UID: \"6ed49186-8fa8-4875-8da3-b74f8fc2b712\") " pod="openshift-multus/multus-additional-cni-plugins-r8fml" Jul 30 19:24:40.375929 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.374800 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/78e98956-00da-43a5-984d-6b90bbc5f2ca-iptables-alerter-script\") pod \"iptables-alerter-dhdl2\" (UID: \"78e98956-00da-43a5-984d-6b90bbc5f2ca\") " pod="openshift-network-operator/iptables-alerter-dhdl2" Jul 30 19:24:40.375929 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.374828 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr5f9\" (UniqueName: \"kubernetes.io/projected/6ed49186-8fa8-4875-8da3-b74f8fc2b712-kube-api-access-dr5f9\") pod \"multus-additional-cni-plugins-r8fml\" (UID: \"6ed49186-8fa8-4875-8da3-b74f8fc2b712\") " pod="openshift-multus/multus-additional-cni-plugins-r8fml" Jul 30 19:24:40.375929 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.374856 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d1e86eb9-30b6-4e63-8046-3867c587e8f9-etc-selinux\") pod \"aws-ebs-csi-driver-node-lp98t\" (UID: \"d1e86eb9-30b6-4e63-8046-3867c587e8f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lp98t" Jul 30 19:24:40.375929 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.374863 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4cdd383f-6493-4d83-80fa-f4ce17e8416e-host-slash\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.376786 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.374968 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d1e86eb9-30b6-4e63-8046-3867c587e8f9-etc-selinux\") pod \"aws-ebs-csi-driver-node-lp98t\" (UID: \"d1e86eb9-30b6-4e63-8046-3867c587e8f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lp98t" Jul 30 19:24:40.376786 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.375014 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4cdd383f-6493-4d83-80fa-f4ce17e8416e-var-lib-openvswitch\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.376786 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.375040 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/47812e97-b92a-460e-8383-6728edc33f73-cni-binary-copy\") pod \"multus-ddrgc\" (UID: \"47812e97-b92a-460e-8383-6728edc33f73\") " pod="openshift-multus/multus-ddrgc" Jul 30 19:24:40.376786 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.375088 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4cdd383f-6493-4d83-80fa-f4ce17e8416e-systemd-units\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.376786 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.375118 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4cdd383f-6493-4d83-80fa-f4ce17e8416e-ovnkube-script-lib\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.376786 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.375134 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6ed49186-8fa8-4875-8da3-b74f8fc2b712-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r8fml\" (UID: \"6ed49186-8fa8-4875-8da3-b74f8fc2b712\") " pod="openshift-multus/multus-additional-cni-plugins-r8fml" Jul 30 19:24:40.376786 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.375125 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4cdd383f-6493-4d83-80fa-f4ce17e8416e-host-cni-netd\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.376786 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.375148 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8f32c8f1-7bf6-4b84-8205-555365c55861-dbus\") pod \"global-pull-secret-syncer-6tpv8\" (UID: \"8f32c8f1-7bf6-4b84-8205-555365c55861\") " pod="kube-system/global-pull-secret-syncer-6tpv8" Jul 30 19:24:40.376786 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.375169 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47812e97-b92a-460e-8383-6728edc33f73-multus-cni-dir\") pod \"multus-ddrgc\" (UID: \"47812e97-b92a-460e-8383-6728edc33f73\") " pod="openshift-multus/multus-ddrgc" Jul 30 19:24:40.376786 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.375199 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/47812e97-b92a-460e-8383-6728edc33f73-host-var-lib-cni-bin\") pod \"multus-ddrgc\" (UID: \"47812e97-b92a-460e-8383-6728edc33f73\") " pod="openshift-multus/multus-ddrgc" Jul 30 19:24:40.376786 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.375208 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6ed49186-8fa8-4875-8da3-b74f8fc2b712-cnibin\") pod \"multus-additional-cni-plugins-r8fml\" (UID: \"6ed49186-8fa8-4875-8da3-b74f8fc2b712\") " pod="openshift-multus/multus-additional-cni-plugins-r8fml" Jul 30 19:24:40.376786 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.375226 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47812e97-b92a-460e-8383-6728edc33f73-etc-kubernetes\") pod \"multus-ddrgc\" (UID: \"47812e97-b92a-460e-8383-6728edc33f73\") " pod="openshift-multus/multus-ddrgc" Jul 30 19:24:40.376786 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.375251 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxr62\" (UniqueName: \"kubernetes.io/projected/47812e97-b92a-460e-8383-6728edc33f73-kube-api-access-rxr62\") pod \"multus-ddrgc\" (UID: \"47812e97-b92a-460e-8383-6728edc33f73\") " pod="openshift-multus/multus-ddrgc" Jul 30 19:24:40.376786 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.375278 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47812e97-b92a-460e-8383-6728edc33f73-multus-cni-dir\") pod \"multus-ddrgc\" (UID: \"47812e97-b92a-460e-8383-6728edc33f73\") " pod="openshift-multus/multus-ddrgc" Jul 30 19:24:40.376786 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.375279 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d1e86eb9-30b6-4e63-8046-3867c587e8f9-sys-fs\") pod \"aws-ebs-csi-driver-node-lp98t\" (UID: \"d1e86eb9-30b6-4e63-8046-3867c587e8f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lp98t" Jul 30 19:24:40.376786 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.375302 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/47812e97-b92a-460e-8383-6728edc33f73-host-var-lib-cni-bin\") pod \"multus-ddrgc\" (UID: \"47812e97-b92a-460e-8383-6728edc33f73\") " pod="openshift-multus/multus-ddrgc" Jul 30 19:24:40.376786 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.375327 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47812e97-b92a-460e-8383-6728edc33f73-etc-kubernetes\") pod \"multus-ddrgc\" (UID: \"47812e97-b92a-460e-8383-6728edc33f73\") " pod="openshift-multus/multus-ddrgc" Jul 30 19:24:40.376786 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.375345 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d1e86eb9-30b6-4e63-8046-3867c587e8f9-sys-fs\") pod \"aws-ebs-csi-driver-node-lp98t\" (UID: \"d1e86eb9-30b6-4e63-8046-3867c587e8f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lp98t" Jul 30 19:24:40.377380 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.375356 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/47812e97-b92a-460e-8383-6728edc33f73-multus-daemon-config\") pod \"multus-ddrgc\" (UID: \"47812e97-b92a-460e-8383-6728edc33f73\") " pod="openshift-multus/multus-ddrgc" Jul 30 19:24:40.377380 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.375391 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/78e98956-00da-43a5-984d-6b90bbc5f2ca-host-slash\") pod \"iptables-alerter-dhdl2\" (UID: \"78e98956-00da-43a5-984d-6b90bbc5f2ca\") " pod="openshift-network-operator/iptables-alerter-dhdl2" Jul 30 19:24:40.377380 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.375419 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8f32c8f1-7bf6-4b84-8205-555365c55861-original-pull-secret\") pod \"global-pull-secret-syncer-6tpv8\" (UID: \"8f32c8f1-7bf6-4b84-8205-555365c55861\") " pod="kube-system/global-pull-secret-syncer-6tpv8" Jul 30 19:24:40.377380 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.375420 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4cdd383f-6493-4d83-80fa-f4ce17e8416e-env-overrides\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.377380 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.375454 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/78e98956-00da-43a5-984d-6b90bbc5f2ca-host-slash\") pod \"iptables-alerter-dhdl2\" (UID: \"78e98956-00da-43a5-984d-6b90bbc5f2ca\") " pod="openshift-network-operator/iptables-alerter-dhdl2" Jul 30 19:24:40.377380 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:40.375530 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 19:24:40.377380 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.375536 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4cdd383f-6493-4d83-80fa-f4ce17e8416e-run-openvswitch\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.377380 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.375565 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4cdd383f-6493-4d83-80fa-f4ce17e8416e-run-ovn\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.377380 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.375591 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ed49186-8fa8-4875-8da3-b74f8fc2b712-system-cni-dir\") pod \"multus-additional-cni-plugins-r8fml\" (UID: \"6ed49186-8fa8-4875-8da3-b74f8fc2b712\") " pod="openshift-multus/multus-additional-cni-plugins-r8fml" Jul 30 19:24:40.377380 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.375608 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4cdd383f-6493-4d83-80fa-f4ce17e8416e-run-openvswitch\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.377380 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.375618 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47812e97-b92a-460e-8383-6728edc33f73-system-cni-dir\") pod \"multus-ddrgc\" (UID: \"47812e97-b92a-460e-8383-6728edc33f73\") " pod="openshift-multus/multus-ddrgc" Jul 30 19:24:40.377380 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.375642 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c058e636-437f-4395-9e4a-760cc5fd051c-host\") pod \"node-ca-rfjmx\" (UID: \"c058e636-437f-4395-9e4a-760cc5fd051c\") " pod="openshift-image-registry/node-ca-rfjmx" Jul 30 19:24:40.377380 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.375652 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4cdd383f-6493-4d83-80fa-f4ce17e8416e-run-ovn\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.377380 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.375670 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c058e636-437f-4395-9e4a-760cc5fd051c-serviceca\") pod \"node-ca-rfjmx\" (UID: \"c058e636-437f-4395-9e4a-760cc5fd051c\") " pod="openshift-image-registry/node-ca-rfjmx" Jul 30 19:24:40.377380 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.375707 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47812e97-b92a-460e-8383-6728edc33f73-system-cni-dir\") pod \"multus-ddrgc\" (UID: \"47812e97-b92a-460e-8383-6728edc33f73\") " pod="openshift-multus/multus-ddrgc" Jul 30 19:24:40.377380 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.375719 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6ed49186-8fa8-4875-8da3-b74f8fc2b712-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r8fml\" (UID: \"6ed49186-8fa8-4875-8da3-b74f8fc2b712\") " pod="openshift-multus/multus-additional-cni-plugins-r8fml" Jul 30 19:24:40.377380 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.375746 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6ed49186-8fa8-4875-8da3-b74f8fc2b712-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r8fml\" (UID: \"6ed49186-8fa8-4875-8da3-b74f8fc2b712\") " pod="openshift-multus/multus-additional-cni-plugins-r8fml" Jul 30 19:24:40.377380 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.375751 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ed49186-8fa8-4875-8da3-b74f8fc2b712-system-cni-dir\") pod \"multus-additional-cni-plugins-r8fml\" (UID: \"6ed49186-8fa8-4875-8da3-b74f8fc2b712\") " pod="openshift-multus/multus-additional-cni-plugins-r8fml" Jul 30 19:24:40.378308 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.375774 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/47812e97-b92a-460e-8383-6728edc33f73-os-release\") pod \"multus-ddrgc\" (UID: \"47812e97-b92a-460e-8383-6728edc33f73\") " pod="openshift-multus/multus-ddrgc" Jul 30 19:24:40.378308 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.375800 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlftb\" (UniqueName: \"kubernetes.io/projected/c058e636-437f-4395-9e4a-760cc5fd051c-kube-api-access-xlftb\") pod \"node-ca-rfjmx\" (UID: \"c058e636-437f-4395-9e4a-760cc5fd051c\") " pod="openshift-image-registry/node-ca-rfjmx" Jul 30 19:24:40.378308 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.375810 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/47812e97-b92a-460e-8383-6728edc33f73-multus-daemon-config\") pod \"multus-ddrgc\" (UID: \"47812e97-b92a-460e-8383-6728edc33f73\") " pod="openshift-multus/multus-ddrgc" Jul 30 19:24:40.378308 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.375826 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grwl9\" (UniqueName: \"kubernetes.io/projected/d1e86eb9-30b6-4e63-8046-3867c587e8f9-kube-api-access-grwl9\") pod \"aws-ebs-csi-driver-node-lp98t\" (UID: \"d1e86eb9-30b6-4e63-8046-3867c587e8f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lp98t" Jul 30 19:24:40.378308 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.375876 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c058e636-437f-4395-9e4a-760cc5fd051c-host\") pod \"node-ca-rfjmx\" (UID: \"c058e636-437f-4395-9e4a-760cc5fd051c\") " pod="openshift-image-registry/node-ca-rfjmx" Jul 30 19:24:40.378308 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.375907 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4cdd383f-6493-4d83-80fa-f4ce17e8416e-host-run-netns\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.378308 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.375951 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4cdd383f-6493-4d83-80fa-f4ce17e8416e-host-run-netns\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.378308 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.375981 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4cdd383f-6493-4d83-80fa-f4ce17e8416e-ovnkube-config\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.378308 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.376010 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6ed49186-8fa8-4875-8da3-b74f8fc2b712-cni-binary-copy\") pod \"multus-additional-cni-plugins-r8fml\" (UID: \"6ed49186-8fa8-4875-8da3-b74f8fc2b712\") " pod="openshift-multus/multus-additional-cni-plugins-r8fml" Jul 30 19:24:40.378308 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.376038 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glnkx\" (UniqueName: \"kubernetes.io/projected/c96c7638-b6ce-4a98-a49d-1bf0fb43edf8-kube-api-access-glnkx\") pod \"network-check-target-czm5g\" (UID: \"c96c7638-b6ce-4a98-a49d-1bf0fb43edf8\") " pod="openshift-network-diagnostics/network-check-target-czm5g" Jul 30 19:24:40.378308 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.376065 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d1e86eb9-30b6-4e63-8046-3867c587e8f9-registration-dir\") pod \"aws-ebs-csi-driver-node-lp98t\" (UID: \"d1e86eb9-30b6-4e63-8046-3867c587e8f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lp98t" Jul 30 19:24:40.378308 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.376093 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/47812e97-b92a-460e-8383-6728edc33f73-hostroot\") pod \"multus-ddrgc\" (UID: \"47812e97-b92a-460e-8383-6728edc33f73\") " pod="openshift-multus/multus-ddrgc" Jul 30 19:24:40.378308 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.376119 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8f32c8f1-7bf6-4b84-8205-555365c55861-kubelet-config\") pod \"global-pull-secret-syncer-6tpv8\" (UID: \"8f32c8f1-7bf6-4b84-8205-555365c55861\") " pod="kube-system/global-pull-secret-syncer-6tpv8" Jul 30 19:24:40.378308 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.376149 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d1e86eb9-30b6-4e63-8046-3867c587e8f9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lp98t\" (UID: \"d1e86eb9-30b6-4e63-8046-3867c587e8f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lp98t" Jul 30 19:24:40.378308 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.376154 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c058e636-437f-4395-9e4a-760cc5fd051c-serviceca\") pod \"node-ca-rfjmx\" (UID: \"c058e636-437f-4395-9e4a-760cc5fd051c\") " pod="openshift-image-registry/node-ca-rfjmx" Jul 30 19:24:40.378308 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.376227 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/47812e97-b92a-460e-8383-6728edc33f73-os-release\") pod \"multus-ddrgc\" (UID: \"47812e97-b92a-460e-8383-6728edc33f73\") " pod="openshift-multus/multus-ddrgc" Jul 30 19:24:40.378308 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.376230 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d1e86eb9-30b6-4e63-8046-3867c587e8f9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lp98t\" (UID: \"d1e86eb9-30b6-4e63-8046-3867c587e8f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lp98t" Jul 30 19:24:40.378930 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.376273 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d1e86eb9-30b6-4e63-8046-3867c587e8f9-registration-dir\") pod \"aws-ebs-csi-driver-node-lp98t\" (UID: \"d1e86eb9-30b6-4e63-8046-3867c587e8f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lp98t" Jul 30 19:24:40.378930 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.376318 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/47812e97-b92a-460e-8383-6728edc33f73-hostroot\") pod \"multus-ddrgc\" (UID: \"47812e97-b92a-460e-8383-6728edc33f73\") " pod="openshift-multus/multus-ddrgc" Jul 30 19:24:40.378930 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.376349 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6ed49186-8fa8-4875-8da3-b74f8fc2b712-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r8fml\" (UID: \"6ed49186-8fa8-4875-8da3-b74f8fc2b712\") " pod="openshift-multus/multus-additional-cni-plugins-r8fml" Jul 30 19:24:40.378930 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.376370 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8f32c8f1-7bf6-4b84-8205-555365c55861-kubelet-config\") pod \"global-pull-secret-syncer-6tpv8\" (UID: \"8f32c8f1-7bf6-4b84-8205-555365c55861\") " pod="kube-system/global-pull-secret-syncer-6tpv8" Jul 30 19:24:40.378930 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.376425 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6ed49186-8fa8-4875-8da3-b74f8fc2b712-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r8fml\" (UID: \"6ed49186-8fa8-4875-8da3-b74f8fc2b712\") " pod="openshift-multus/multus-additional-cni-plugins-r8fml" Jul 30 19:24:40.378930 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:40.376694 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f32c8f1-7bf6-4b84-8205-555365c55861-original-pull-secret podName:8f32c8f1-7bf6-4b84-8205-555365c55861 nodeName:}" failed. No retries permitted until 2026-07-30 19:24:40.876673726 +0000 UTC m=+3.096232632 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8f32c8f1-7bf6-4b84-8205-555365c55861-original-pull-secret") pod "global-pull-secret-syncer-6tpv8" (UID: "8f32c8f1-7bf6-4b84-8205-555365c55861") : object "kube-system"/"original-pull-secret" not registered Jul 30 19:24:40.378930 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.376784 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4cdd383f-6493-4d83-80fa-f4ce17e8416e-ovnkube-config\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.378930 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.376798 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6ed49186-8fa8-4875-8da3-b74f8fc2b712-cni-binary-copy\") pod \"multus-additional-cni-plugins-r8fml\" (UID: \"6ed49186-8fa8-4875-8da3-b74f8fc2b712\") " pod="openshift-multus/multus-additional-cni-plugins-r8fml" Jul 30 19:24:40.378930 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.376876 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fb61e5ec-e983-407c-962c-1552688cbe5c-agent-certs\") pod \"konnectivity-agent-ttglk\" (UID: \"fb61e5ec-e983-407c-962c-1552688cbe5c\") " pod="kube-system/konnectivity-agent-ttglk" Jul 30 19:24:40.378930 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.377868 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4cdd383f-6493-4d83-80fa-f4ce17e8416e-ovn-node-metrics-cert\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.390751 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:40.390109 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 19:24:40.390751 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:40.390135 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 19:24:40.390751 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:40.390149 2556 projected.go:196] Error preparing data for projected volume kube-api-access-glnkx for pod openshift-network-diagnostics/network-check-target-czm5g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 19:24:40.390751 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:40.390209 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c96c7638-b6ce-4a98-a49d-1bf0fb43edf8-kube-api-access-glnkx podName:c96c7638-b6ce-4a98-a49d-1bf0fb43edf8 nodeName:}" failed. No retries permitted until 2026-07-30 19:24:40.890193205 +0000 UTC m=+3.109752099 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-glnkx" (UniqueName: "kubernetes.io/projected/c96c7638-b6ce-4a98-a49d-1bf0fb43edf8-kube-api-access-glnkx") pod "network-check-target-czm5g" (UID: "c96c7638-b6ce-4a98-a49d-1bf0fb43edf8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 19:24:40.393008 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.392953 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxr62\" (UniqueName: \"kubernetes.io/projected/47812e97-b92a-460e-8383-6728edc33f73-kube-api-access-rxr62\") pod \"multus-ddrgc\" (UID: \"47812e97-b92a-460e-8383-6728edc33f73\") " pod="openshift-multus/multus-ddrgc" Jul 30 19:24:40.393773 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.393747 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl6vj\" (UniqueName: \"kubernetes.io/projected/9547ff50-df97-486d-85eb-c253420d0e73-kube-api-access-xl6vj\") pod \"network-metrics-daemon-vxwhr\" (UID: \"9547ff50-df97-486d-85eb-c253420d0e73\") " pod="openshift-multus/network-metrics-daemon-vxwhr" Jul 30 19:24:40.394674 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.394544 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr5f9\" (UniqueName: \"kubernetes.io/projected/6ed49186-8fa8-4875-8da3-b74f8fc2b712-kube-api-access-dr5f9\") pod \"multus-additional-cni-plugins-r8fml\" (UID: \"6ed49186-8fa8-4875-8da3-b74f8fc2b712\") " pod="openshift-multus/multus-additional-cni-plugins-r8fml" Jul 30 19:24:40.394674 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.394598 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grwl9\" (UniqueName: \"kubernetes.io/projected/d1e86eb9-30b6-4e63-8046-3867c587e8f9-kube-api-access-grwl9\") pod \"aws-ebs-csi-driver-node-lp98t\" (UID: \"d1e86eb9-30b6-4e63-8046-3867c587e8f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lp98t" Jul 30 19:24:40.394867 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.394679 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxfgl\" (UniqueName: \"kubernetes.io/projected/78e98956-00da-43a5-984d-6b90bbc5f2ca-kube-api-access-cxfgl\") pod \"iptables-alerter-dhdl2\" (UID: \"78e98956-00da-43a5-984d-6b90bbc5f2ca\") " pod="openshift-network-operator/iptables-alerter-dhdl2" Jul 30 19:24:40.395128 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.395085 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlftb\" (UniqueName: \"kubernetes.io/projected/c058e636-437f-4395-9e4a-760cc5fd051c-kube-api-access-xlftb\") pod \"node-ca-rfjmx\" (UID: \"c058e636-437f-4395-9e4a-760cc5fd051c\") " pod="openshift-image-registry/node-ca-rfjmx" Jul 30 19:24:40.395302 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.395276 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdx6n\" (UniqueName: \"kubernetes.io/projected/4cdd383f-6493-4d83-80fa-f4ce17e8416e-kube-api-access-xdx6n\") pod \"ovnkube-node-795qw\" (UID: \"4cdd383f-6493-4d83-80fa-f4ce17e8416e\") " pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.469604 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.469573 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5p5fm" Jul 30 19:24:40.483076 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.483042 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qjcpf" Jul 30 19:24:40.494651 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.494626 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rfjmx" Jul 30 19:24:40.508431 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.508407 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:24:40.516121 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.516099 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-r8fml" Jul 30 19:24:40.526003 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.525954 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ddrgc" Jul 30 19:24:40.534273 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.534247 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ttglk" Jul 30 19:24:40.544358 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.544324 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dhdl2" Jul 30 19:24:40.551256 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.551237 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lp98t" Jul 30 19:24:40.879988 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.879886 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8f32c8f1-7bf6-4b84-8205-555365c55861-original-pull-secret\") pod \"global-pull-secret-syncer-6tpv8\" (UID: \"8f32c8f1-7bf6-4b84-8205-555365c55861\") " pod="kube-system/global-pull-secret-syncer-6tpv8" Jul 30 19:24:40.879988 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.879974 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9547ff50-df97-486d-85eb-c253420d0e73-metrics-certs\") pod \"network-metrics-daemon-vxwhr\" (UID: \"9547ff50-df97-486d-85eb-c253420d0e73\") " pod="openshift-multus/network-metrics-daemon-vxwhr" Jul 30 19:24:40.880211 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:40.880069 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 19:24:40.880211 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:40.880130 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f32c8f1-7bf6-4b84-8205-555365c55861-original-pull-secret podName:8f32c8f1-7bf6-4b84-8205-555365c55861 nodeName:}" failed. No retries permitted until 2026-07-30 19:24:41.880115795 +0000 UTC m=+4.099674690 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8f32c8f1-7bf6-4b84-8205-555365c55861-original-pull-secret") pod "global-pull-secret-syncer-6tpv8" (UID: "8f32c8f1-7bf6-4b84-8205-555365c55861") : object "kube-system"/"original-pull-secret" not registered Jul 30 19:24:40.880211 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:40.880069 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 19:24:40.880211 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:40.880207 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9547ff50-df97-486d-85eb-c253420d0e73-metrics-certs podName:9547ff50-df97-486d-85eb-c253420d0e73 nodeName:}" failed. No retries permitted until 2026-07-30 19:24:41.880195256 +0000 UTC m=+4.099754162 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9547ff50-df97-486d-85eb-c253420d0e73-metrics-certs") pod "network-metrics-daemon-vxwhr" (UID: "9547ff50-df97-486d-85eb-c253420d0e73") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 19:24:40.980429 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:40.980383 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glnkx\" (UniqueName: \"kubernetes.io/projected/c96c7638-b6ce-4a98-a49d-1bf0fb43edf8-kube-api-access-glnkx\") pod \"network-check-target-czm5g\" (UID: \"c96c7638-b6ce-4a98-a49d-1bf0fb43edf8\") " pod="openshift-network-diagnostics/network-check-target-czm5g" Jul 30 19:24:40.980621 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:40.980523 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 19:24:40.980621 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:40.980549 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 19:24:40.980621 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:40.980562 2556 projected.go:196] Error preparing data for projected volume kube-api-access-glnkx for pod openshift-network-diagnostics/network-check-target-czm5g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 19:24:40.980789 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:40.980630 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c96c7638-b6ce-4a98-a49d-1bf0fb43edf8-kube-api-access-glnkx podName:c96c7638-b6ce-4a98-a49d-1bf0fb43edf8 nodeName:}" failed. No retries permitted until 2026-07-30 19:24:41.980611243 +0000 UTC m=+4.200170135 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-glnkx" (UniqueName: "kubernetes.io/projected/c96c7638-b6ce-4a98-a49d-1bf0fb43edf8-kube-api-access-glnkx") pod "network-check-target-czm5g" (UID: "c96c7638-b6ce-4a98-a49d-1bf0fb43edf8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 19:24:41.201105 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:41.201021 2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-29 19:19:40 +0000 UTC" deadline="2027-12-30 19:06:54.893344959 +0000 UTC" Jul 30 19:24:41.201105 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:41.201061 2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12431h42m13.69228864s" Jul 30 19:24:41.318667 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:41.318634 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxwhr" Jul 30 19:24:41.318825 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:41.318746 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxwhr" podUID="9547ff50-df97-486d-85eb-c253420d0e73" Jul 30 19:24:41.456947 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:41.456921 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc058e636_437f_4395_9e4a_760cc5fd051c.slice/crio-f3601a80c376f129f128cc985740d49baca7153453c482a1718f06c8a8ab9a41 WatchSource:0}: Error finding container f3601a80c376f129f128cc985740d49baca7153453c482a1718f06c8a8ab9a41: Status 404 returned error can't find the container with id f3601a80c376f129f128cc985740d49baca7153453c482a1718f06c8a8ab9a41 Jul 30 19:24:41.470400 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:41.470345 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47812e97_b92a_460e_8383_6728edc33f73.slice/crio-bcd9711a3555a08df0b47e8cae0edd78a50c7d15679e6df0181819e006464336 WatchSource:0}: Error finding container bcd9711a3555a08df0b47e8cae0edd78a50c7d15679e6df0181819e006464336: Status 404 returned error can't find the container with id bcd9711a3555a08df0b47e8cae0edd78a50c7d15679e6df0181819e006464336 Jul 30 19:24:41.473328 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:41.473293 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed49186_8fa8_4875_8da3_b74f8fc2b712.slice/crio-22b576181f6b211c865a93e9ae7f19f39eb44808c586097e7ac41506ccc44bec WatchSource:0}: Error finding container 22b576181f6b211c865a93e9ae7f19f39eb44808c586097e7ac41506ccc44bec: Status 404 returned error can't find the container with id 22b576181f6b211c865a93e9ae7f19f39eb44808c586097e7ac41506ccc44bec Jul 30 19:24:41.473976 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:41.473934 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9a5febb_adab_42b2_a740_a057ada6294a.slice/crio-88bd669305d134358a1f2d113260c22463a502dd3623f777667355cbe41171fd WatchSource:0}: Error finding container 88bd669305d134358a1f2d113260c22463a502dd3623f777667355cbe41171fd: Status 404 returned error can't find the container with id 88bd669305d134358a1f2d113260c22463a502dd3623f777667355cbe41171fd Jul 30 19:24:41.474668 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:24:41.474643 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1e86eb9_30b6_4e63_8046_3867c587e8f9.slice/crio-ff6e02a28a19e44f801d3afe3fd1476b30a44c3a2a45bb207c57ad6656ae3b97 WatchSource:0}: Error finding container ff6e02a28a19e44f801d3afe3fd1476b30a44c3a2a45bb207c57ad6656ae3b97: Status 404 returned error can't find the container with id ff6e02a28a19e44f801d3afe3fd1476b30a44c3a2a45bb207c57ad6656ae3b97 Jul 30 19:24:41.888072 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:41.887862 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8f32c8f1-7bf6-4b84-8205-555365c55861-original-pull-secret\") pod \"global-pull-secret-syncer-6tpv8\" (UID: \"8f32c8f1-7bf6-4b84-8205-555365c55861\") " pod="kube-system/global-pull-secret-syncer-6tpv8" Jul 30 19:24:41.888268 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:41.888127 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9547ff50-df97-486d-85eb-c253420d0e73-metrics-certs\") pod \"network-metrics-daemon-vxwhr\" (UID: \"9547ff50-df97-486d-85eb-c253420d0e73\") " pod="openshift-multus/network-metrics-daemon-vxwhr" Jul 30 19:24:41.888268 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:41.888013 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 19:24:41.888268 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:41.888235 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 19:24:41.888268 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:41.888262 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f32c8f1-7bf6-4b84-8205-555365c55861-original-pull-secret podName:8f32c8f1-7bf6-4b84-8205-555365c55861 nodeName:}" failed. No retries permitted until 2026-07-30 19:24:43.888233125 +0000 UTC m=+6.107792036 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8f32c8f1-7bf6-4b84-8205-555365c55861-original-pull-secret") pod "global-pull-secret-syncer-6tpv8" (UID: "8f32c8f1-7bf6-4b84-8205-555365c55861") : object "kube-system"/"original-pull-secret" not registered Jul 30 19:24:41.888516 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:41.888291 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9547ff50-df97-486d-85eb-c253420d0e73-metrics-certs podName:9547ff50-df97-486d-85eb-c253420d0e73 nodeName:}" failed. No retries permitted until 2026-07-30 19:24:43.888273076 +0000 UTC m=+6.107831970 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9547ff50-df97-486d-85eb-c253420d0e73-metrics-certs") pod "network-metrics-daemon-vxwhr" (UID: "9547ff50-df97-486d-85eb-c253420d0e73") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 19:24:41.988728 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:41.988648 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glnkx\" (UniqueName: \"kubernetes.io/projected/c96c7638-b6ce-4a98-a49d-1bf0fb43edf8-kube-api-access-glnkx\") pod \"network-check-target-czm5g\" (UID: \"c96c7638-b6ce-4a98-a49d-1bf0fb43edf8\") " pod="openshift-network-diagnostics/network-check-target-czm5g" Jul 30 19:24:41.988896 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:41.988858 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 19:24:41.988967 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:41.988910 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 19:24:41.988967 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:41.988924 2556 projected.go:196] Error preparing data for projected volume kube-api-access-glnkx for pod openshift-network-diagnostics/network-check-target-czm5g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 19:24:41.989074 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:41.988984 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c96c7638-b6ce-4a98-a49d-1bf0fb43edf8-kube-api-access-glnkx podName:c96c7638-b6ce-4a98-a49d-1bf0fb43edf8 nodeName:}" failed. No retries permitted until 2026-07-30 19:24:43.988963935 +0000 UTC m=+6.208522830 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-glnkx" (UniqueName: "kubernetes.io/projected/c96c7638-b6ce-4a98-a49d-1bf0fb43edf8-kube-api-access-glnkx") pod "network-check-target-czm5g" (UID: "c96c7638-b6ce-4a98-a49d-1bf0fb43edf8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 19:24:42.201597 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:42.201528 2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-29 19:19:40 +0000 UTC" deadline="2028-04-19 12:39:43.55028103 +0000 UTC" Jul 30 19:24:42.201597 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:42.201596 2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15089h15m1.348689817s" Jul 30 19:24:42.320319 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:42.319572 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-czm5g" Jul 30 19:24:42.320319 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:42.319708 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-czm5g" podUID="c96c7638-b6ce-4a98-a49d-1bf0fb43edf8" Jul 30 19:24:42.320806 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:42.320638 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tpv8" Jul 30 19:24:42.320806 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:42.320749 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6tpv8" podUID="8f32c8f1-7bf6-4b84-8205-555365c55861" Jul 30 19:24:42.344452 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:42.344367 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lp98t" event={"ID":"d1e86eb9-30b6-4e63-8046-3867c587e8f9","Type":"ContainerStarted","Data":"ff6e02a28a19e44f801d3afe3fd1476b30a44c3a2a45bb207c57ad6656ae3b97"} Jul 30 19:24:42.345966 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:42.345895 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5p5fm" event={"ID":"e9a5febb-adab-42b2-a740-a057ada6294a","Type":"ContainerStarted","Data":"88bd669305d134358a1f2d113260c22463a502dd3623f777667355cbe41171fd"} Jul 30 19:24:42.351307 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:42.351281 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r8fml" event={"ID":"6ed49186-8fa8-4875-8da3-b74f8fc2b712","Type":"ContainerStarted","Data":"22b576181f6b211c865a93e9ae7f19f39eb44808c586097e7ac41506ccc44bec"} Jul 30 19:24:42.353699 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:42.353672 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rfjmx" event={"ID":"c058e636-437f-4395-9e4a-760cc5fd051c","Type":"ContainerStarted","Data":"f3601a80c376f129f128cc985740d49baca7153453c482a1718f06c8a8ab9a41"} Jul 30 19:24:42.362637 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:42.362601 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dhdl2" event={"ID":"78e98956-00da-43a5-984d-6b90bbc5f2ca","Type":"ContainerStarted","Data":"efaeef5558184949e97b72fdc3b2a69407623333cd647e851af51feb3516a237"} Jul 30 19:24:42.366676 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:42.366650 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qjcpf" event={"ID":"87b39982-2538-4dcf-baa2-6c522b0c4896","Type":"ContainerStarted","Data":"ba93013b3d1f09fb3dbe795c1c4df94d3f21a3918ee2dd5893d5e4756c4e5794"} Jul 30 19:24:42.371641 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:42.371617 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ddrgc" event={"ID":"47812e97-b92a-460e-8383-6728edc33f73","Type":"ContainerStarted","Data":"bcd9711a3555a08df0b47e8cae0edd78a50c7d15679e6df0181819e006464336"} Jul 30 19:24:42.375767 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:42.375426 2556 generic.go:356] "Generic (PLEG): container finished" podID="2af0cc64bc27103c007e4f088d273813" containerID="f810f787488b5ecc0e5200c5d3ca33e91542fd88377daba056298e508b109fe7" exitCode=0 Jul 30 19:24:42.375767 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:42.375519 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal" event={"ID":"2af0cc64bc27103c007e4f088d273813","Type":"ContainerDied","Data":"f810f787488b5ecc0e5200c5d3ca33e91542fd88377daba056298e508b109fe7"} Jul 30 19:24:42.378609 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:42.378498 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-105.ec2.internal" event={"ID":"1728156c8efd1097c5a710ba4d71fa3e","Type":"ContainerStarted","Data":"eb1230562b1cfac335910565d3b0ed3b4659f3116fdfb3a628f0a2dcd2576f5f"} Jul 30 19:24:42.389624 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:42.388246 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ttglk" event={"ID":"fb61e5ec-e983-407c-962c-1552688cbe5c","Type":"ContainerStarted","Data":"e07b991baf096fd1ed318af410ad0737569b24e9ad3ee86f63696c7f3316f58c"} Jul 30 19:24:42.400816 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:42.400789 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-795qw" event={"ID":"4cdd383f-6493-4d83-80fa-f4ce17e8416e","Type":"ContainerStarted","Data":"d78266ccf3a128d809d48818b9c68e683f90cc5fc0313985b501558c6f7dc7bd"} Jul 30 19:24:43.318694 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:43.318660 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxwhr" Jul 30 19:24:43.319192 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:43.318806 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxwhr" podUID="9547ff50-df97-486d-85eb-c253420d0e73" Jul 30 19:24:43.414692 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:43.414548 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal" event={"ID":"2af0cc64bc27103c007e4f088d273813","Type":"ContainerStarted","Data":"30f9ea0a373f804dfc4a04c13d40b59ae03dd1ffd9f6ff98663b075bf3de5f9c"} Jul 30 19:24:43.430947 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:43.429970 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-105.ec2.internal" podStartSLOduration=4.429950466 podStartE2EDuration="4.429950466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 19:24:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:24:43.429534904 +0000 UTC m=+5.649093836" watchObservedRunningTime="2026-07-30 19:24:43.429950466 +0000 UTC m=+5.649509383" Jul 30 19:24:43.430947 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:43.430434 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-105.ec2.internal" podStartSLOduration=4.430421835 podStartE2EDuration="4.430421835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 19:24:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:24:42.412152443 +0000 UTC m=+4.631711360" watchObservedRunningTime="2026-07-30 19:24:43.430421835 +0000 UTC m=+5.649980751" Jul 30 19:24:43.905913 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:43.905872 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9547ff50-df97-486d-85eb-c253420d0e73-metrics-certs\") pod \"network-metrics-daemon-vxwhr\" (UID: \"9547ff50-df97-486d-85eb-c253420d0e73\") " pod="openshift-multus/network-metrics-daemon-vxwhr" Jul 30 19:24:43.906119 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:43.905949 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8f32c8f1-7bf6-4b84-8205-555365c55861-original-pull-secret\") pod \"global-pull-secret-syncer-6tpv8\" (UID: \"8f32c8f1-7bf6-4b84-8205-555365c55861\") " pod="kube-system/global-pull-secret-syncer-6tpv8" Jul 30 19:24:43.906119 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:43.906114 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 19:24:43.906235 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:43.906180 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f32c8f1-7bf6-4b84-8205-555365c55861-original-pull-secret podName:8f32c8f1-7bf6-4b84-8205-555365c55861 nodeName:}" failed. No retries permitted until 2026-07-30 19:24:47.906161164 +0000 UTC m=+10.125720062 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8f32c8f1-7bf6-4b84-8205-555365c55861-original-pull-secret") pod "global-pull-secret-syncer-6tpv8" (UID: "8f32c8f1-7bf6-4b84-8205-555365c55861") : object "kube-system"/"original-pull-secret" not registered Jul 30 19:24:43.906635 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:43.906607 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 19:24:43.906752 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:43.906655 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9547ff50-df97-486d-85eb-c253420d0e73-metrics-certs podName:9547ff50-df97-486d-85eb-c253420d0e73 nodeName:}" failed. No retries permitted until 2026-07-30 19:24:47.90664043 +0000 UTC m=+10.126199342 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9547ff50-df97-486d-85eb-c253420d0e73-metrics-certs") pod "network-metrics-daemon-vxwhr" (UID: "9547ff50-df97-486d-85eb-c253420d0e73") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 19:24:44.006611 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:44.006571 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glnkx\" (UniqueName: \"kubernetes.io/projected/c96c7638-b6ce-4a98-a49d-1bf0fb43edf8-kube-api-access-glnkx\") pod \"network-check-target-czm5g\" (UID: \"c96c7638-b6ce-4a98-a49d-1bf0fb43edf8\") " pod="openshift-network-diagnostics/network-check-target-czm5g" Jul 30 19:24:44.006792 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:44.006754 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 19:24:44.006792 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:44.006775 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 19:24:44.006792 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:44.006789 2556 projected.go:196] Error preparing data for projected volume kube-api-access-glnkx for pod openshift-network-diagnostics/network-check-target-czm5g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 19:24:44.006965 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:44.006848 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c96c7638-b6ce-4a98-a49d-1bf0fb43edf8-kube-api-access-glnkx podName:c96c7638-b6ce-4a98-a49d-1bf0fb43edf8 nodeName:}" failed. No retries permitted until 2026-07-30 19:24:48.006830271 +0000 UTC m=+10.226389169 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-glnkx" (UniqueName: "kubernetes.io/projected/c96c7638-b6ce-4a98-a49d-1bf0fb43edf8-kube-api-access-glnkx") pod "network-check-target-czm5g" (UID: "c96c7638-b6ce-4a98-a49d-1bf0fb43edf8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 19:24:44.319644 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:44.319559 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-czm5g" Jul 30 19:24:44.320115 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:44.319715 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-czm5g" podUID="c96c7638-b6ce-4a98-a49d-1bf0fb43edf8" Jul 30 19:24:44.320115 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:44.319764 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tpv8" Jul 30 19:24:44.330099 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:44.330057 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6tpv8" podUID="8f32c8f1-7bf6-4b84-8205-555365c55861" Jul 30 19:24:45.318541 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:45.318015 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxwhr" Jul 30 19:24:45.318541 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:45.318165 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxwhr" podUID="9547ff50-df97-486d-85eb-c253420d0e73" Jul 30 19:24:46.319222 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:46.319183 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-czm5g" Jul 30 19:24:46.319222 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:46.319222 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tpv8" Jul 30 19:24:46.319715 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:46.319294 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-czm5g" podUID="c96c7638-b6ce-4a98-a49d-1bf0fb43edf8" Jul 30 19:24:46.319715 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:46.319418 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6tpv8" podUID="8f32c8f1-7bf6-4b84-8205-555365c55861" Jul 30 19:24:47.318095 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:47.318070 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxwhr" Jul 30 19:24:47.318287 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:47.318181 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxwhr" podUID="9547ff50-df97-486d-85eb-c253420d0e73" Jul 30 19:24:47.947356 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:47.947320 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9547ff50-df97-486d-85eb-c253420d0e73-metrics-certs\") pod \"network-metrics-daemon-vxwhr\" (UID: \"9547ff50-df97-486d-85eb-c253420d0e73\") " pod="openshift-multus/network-metrics-daemon-vxwhr" Jul 30 19:24:47.947834 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:47.947404 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8f32c8f1-7bf6-4b84-8205-555365c55861-original-pull-secret\") pod \"global-pull-secret-syncer-6tpv8\" (UID: \"8f32c8f1-7bf6-4b84-8205-555365c55861\") " pod="kube-system/global-pull-secret-syncer-6tpv8" Jul 30 19:24:47.947834 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:47.947540 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 19:24:47.947834 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:47.947598 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f32c8f1-7bf6-4b84-8205-555365c55861-original-pull-secret podName:8f32c8f1-7bf6-4b84-8205-555365c55861 nodeName:}" failed. No retries permitted until 2026-07-30 19:24:55.947581117 +0000 UTC m=+18.167140007 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8f32c8f1-7bf6-4b84-8205-555365c55861-original-pull-secret") pod "global-pull-secret-syncer-6tpv8" (UID: "8f32c8f1-7bf6-4b84-8205-555365c55861") : object "kube-system"/"original-pull-secret" not registered Jul 30 19:24:47.947994 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:47.947912 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 19:24:47.947994 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:47.947953 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9547ff50-df97-486d-85eb-c253420d0e73-metrics-certs podName:9547ff50-df97-486d-85eb-c253420d0e73 nodeName:}" failed. No retries permitted until 2026-07-30 19:24:55.947937829 +0000 UTC m=+18.167496720 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9547ff50-df97-486d-85eb-c253420d0e73-metrics-certs") pod "network-metrics-daemon-vxwhr" (UID: "9547ff50-df97-486d-85eb-c253420d0e73") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 19:24:48.047885 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:48.047850 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glnkx\" (UniqueName: \"kubernetes.io/projected/c96c7638-b6ce-4a98-a49d-1bf0fb43edf8-kube-api-access-glnkx\") pod \"network-check-target-czm5g\" (UID: \"c96c7638-b6ce-4a98-a49d-1bf0fb43edf8\") " pod="openshift-network-diagnostics/network-check-target-czm5g" Jul 30 19:24:48.048067 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:48.047995 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 19:24:48.048067 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:48.048013 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 19:24:48.048067 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:48.048022 2556 projected.go:196] Error preparing data for projected volume kube-api-access-glnkx for pod openshift-network-diagnostics/network-check-target-czm5g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 19:24:48.048220 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:48.048084 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c96c7638-b6ce-4a98-a49d-1bf0fb43edf8-kube-api-access-glnkx podName:c96c7638-b6ce-4a98-a49d-1bf0fb43edf8 nodeName:}" failed. No retries permitted until 2026-07-30 19:24:56.048064074 +0000 UTC m=+18.267622984 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-glnkx" (UniqueName: "kubernetes.io/projected/c96c7638-b6ce-4a98-a49d-1bf0fb43edf8-kube-api-access-glnkx") pod "network-check-target-czm5g" (UID: "c96c7638-b6ce-4a98-a49d-1bf0fb43edf8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 19:24:48.318923 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:48.318850 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-czm5g" Jul 30 19:24:48.319080 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:48.318960 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-czm5g" podUID="c96c7638-b6ce-4a98-a49d-1bf0fb43edf8" Jul 30 19:24:48.319080 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:48.318971 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tpv8" Jul 30 19:24:48.319080 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:48.319041 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6tpv8" podUID="8f32c8f1-7bf6-4b84-8205-555365c55861" Jul 30 19:24:49.318556 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:49.318477 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxwhr" Jul 30 19:24:49.318988 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:49.318633 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxwhr" podUID="9547ff50-df97-486d-85eb-c253420d0e73" Jul 30 19:24:50.317978 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:50.317948 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tpv8" Jul 30 19:24:50.318142 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:50.318080 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6tpv8" podUID="8f32c8f1-7bf6-4b84-8205-555365c55861" Jul 30 19:24:50.318225 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:50.318142 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-czm5g" Jul 30 19:24:50.318312 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:50.318288 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-czm5g" podUID="c96c7638-b6ce-4a98-a49d-1bf0fb43edf8" Jul 30 19:24:51.317959 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:51.317926 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxwhr" Jul 30 19:24:51.318376 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:51.318057 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxwhr" podUID="9547ff50-df97-486d-85eb-c253420d0e73" Jul 30 19:24:52.318472 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:52.318444 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-czm5g" Jul 30 19:24:52.318892 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:52.318564 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-czm5g" podUID="c96c7638-b6ce-4a98-a49d-1bf0fb43edf8" Jul 30 19:24:52.318892 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:52.318646 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tpv8" Jul 30 19:24:52.318892 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:52.318756 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6tpv8" podUID="8f32c8f1-7bf6-4b84-8205-555365c55861" Jul 30 19:24:53.318178 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:53.318148 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxwhr" Jul 30 19:24:53.318357 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:53.318283 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxwhr" podUID="9547ff50-df97-486d-85eb-c253420d0e73" Jul 30 19:24:54.317853 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:54.317819 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-czm5g" Jul 30 19:24:54.318278 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:54.317947 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-czm5g" podUID="c96c7638-b6ce-4a98-a49d-1bf0fb43edf8" Jul 30 19:24:54.318278 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:54.317982 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tpv8" Jul 30 19:24:54.318278 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:54.318058 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6tpv8" podUID="8f32c8f1-7bf6-4b84-8205-555365c55861" Jul 30 19:24:55.317952 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:55.317925 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxwhr" Jul 30 19:24:55.318349 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:55.318021 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxwhr" podUID="9547ff50-df97-486d-85eb-c253420d0e73" Jul 30 19:24:56.012459 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:56.012365 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9547ff50-df97-486d-85eb-c253420d0e73-metrics-certs\") pod \"network-metrics-daemon-vxwhr\" (UID: \"9547ff50-df97-486d-85eb-c253420d0e73\") " pod="openshift-multus/network-metrics-daemon-vxwhr" Jul 30 19:24:56.012459 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:56.012438 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8f32c8f1-7bf6-4b84-8205-555365c55861-original-pull-secret\") pod \"global-pull-secret-syncer-6tpv8\" (UID: \"8f32c8f1-7bf6-4b84-8205-555365c55861\") " pod="kube-system/global-pull-secret-syncer-6tpv8" Jul 30 19:24:56.012645 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:56.012543 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 19:24:56.012645 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:56.012615 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9547ff50-df97-486d-85eb-c253420d0e73-metrics-certs podName:9547ff50-df97-486d-85eb-c253420d0e73 nodeName:}" failed. No retries permitted until 2026-07-30 19:25:12.012594773 +0000 UTC m=+34.232153678 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9547ff50-df97-486d-85eb-c253420d0e73-metrics-certs") pod "network-metrics-daemon-vxwhr" (UID: "9547ff50-df97-486d-85eb-c253420d0e73") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 19:24:56.012645 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:56.012550 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 19:24:56.012846 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:56.012702 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f32c8f1-7bf6-4b84-8205-555365c55861-original-pull-secret podName:8f32c8f1-7bf6-4b84-8205-555365c55861 nodeName:}" failed. No retries permitted until 2026-07-30 19:25:12.012684376 +0000 UTC m=+34.232243271 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8f32c8f1-7bf6-4b84-8205-555365c55861-original-pull-secret") pod "global-pull-secret-syncer-6tpv8" (UID: "8f32c8f1-7bf6-4b84-8205-555365c55861") : object "kube-system"/"original-pull-secret" not registered Jul 30 19:24:56.112829 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:56.112797 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glnkx\" (UniqueName: \"kubernetes.io/projected/c96c7638-b6ce-4a98-a49d-1bf0fb43edf8-kube-api-access-glnkx\") pod \"network-check-target-czm5g\" (UID: \"c96c7638-b6ce-4a98-a49d-1bf0fb43edf8\") " pod="openshift-network-diagnostics/network-check-target-czm5g" Jul 30 19:24:56.112963 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:56.112926 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 19:24:56.112963 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:56.112942 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 19:24:56.112963 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:56.112951 2556 projected.go:196] Error preparing data for projected volume kube-api-access-glnkx for pod openshift-network-diagnostics/network-check-target-czm5g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 19:24:56.113089 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:56.112992 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c96c7638-b6ce-4a98-a49d-1bf0fb43edf8-kube-api-access-glnkx podName:c96c7638-b6ce-4a98-a49d-1bf0fb43edf8 nodeName:}" failed. No retries permitted until 2026-07-30 19:25:12.112979698 +0000 UTC m=+34.332538588 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-glnkx" (UniqueName: "kubernetes.io/projected/c96c7638-b6ce-4a98-a49d-1bf0fb43edf8-kube-api-access-glnkx") pod "network-check-target-czm5g" (UID: "c96c7638-b6ce-4a98-a49d-1bf0fb43edf8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 19:24:56.318480 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:56.318395 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-czm5g" Jul 30 19:24:56.318904 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:56.318523 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-czm5g" podUID="c96c7638-b6ce-4a98-a49d-1bf0fb43edf8" Jul 30 19:24:56.318904 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:56.318561 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tpv8" Jul 30 19:24:56.318904 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:56.318621 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6tpv8" podUID="8f32c8f1-7bf6-4b84-8205-555365c55861" Jul 30 19:24:57.318682 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:57.318652 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxwhr" Jul 30 19:24:57.319092 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:57.318782 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxwhr" podUID="9547ff50-df97-486d-85eb-c253420d0e73" Jul 30 19:24:58.318422 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:58.318376 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-czm5g" Jul 30 19:24:58.318575 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:58.318488 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tpv8" Jul 30 19:24:58.318575 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:58.318501 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-czm5g" podUID="c96c7638-b6ce-4a98-a49d-1bf0fb43edf8" Jul 30 19:24:58.318575 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:58.318559 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6tpv8" podUID="8f32c8f1-7bf6-4b84-8205-555365c55861" Jul 30 19:24:59.318322 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:24:59.318293 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxwhr" Jul 30 19:24:59.318781 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:24:59.318427 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxwhr" podUID="9547ff50-df97-486d-85eb-c253420d0e73" Jul 30 19:25:00.318665 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:00.318506 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-czm5g" Jul 30 19:25:00.319193 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:00.318731 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-czm5g" podUID="c96c7638-b6ce-4a98-a49d-1bf0fb43edf8" Jul 30 19:25:00.319193 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:00.318591 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tpv8" Jul 30 19:25:00.319193 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:00.318787 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6tpv8" podUID="8f32c8f1-7bf6-4b84-8205-555365c55861" Jul 30 19:25:00.444989 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:00.444965 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ttglk" event={"ID":"fb61e5ec-e983-407c-962c-1552688cbe5c","Type":"ContainerStarted","Data":"2525814ce4a8eaed908c971e63abe82c161b44d21a2a0823880f2263855f0f74"} Jul 30 19:25:00.446826 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:00.446808 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-795qw_4cdd383f-6493-4d83-80fa-f4ce17e8416e/ovn-acl-logging/0.log" Jul 30 19:25:00.447228 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:00.447201 2556 generic.go:356] "Generic (PLEG): container finished" podID="4cdd383f-6493-4d83-80fa-f4ce17e8416e" containerID="b8fd40b133ae60f84f8be5ed5ed4460c7da14093366f7efdcd1c26cd7c15fc32" exitCode=1 Jul 30 19:25:00.447310 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:00.447250 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-795qw" event={"ID":"4cdd383f-6493-4d83-80fa-f4ce17e8416e","Type":"ContainerDied","Data":"b8fd40b133ae60f84f8be5ed5ed4460c7da14093366f7efdcd1c26cd7c15fc32"} Jul 30 19:25:00.447369 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:00.447327 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-795qw" event={"ID":"4cdd383f-6493-4d83-80fa-f4ce17e8416e","Type":"ContainerStarted","Data":"f767a8300d4348da727a0baaa09ff6b59eb9cb3f8fa9a83543a122f4c8a2282f"} Jul 30 19:25:00.448812 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:00.448786 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lp98t" event={"ID":"d1e86eb9-30b6-4e63-8046-3867c587e8f9","Type":"ContainerStarted","Data":"698e65010d750fd613b1b6592b339587981f7a9111c60a58d2d1aa6551709e97"} Jul 30 19:25:00.450156 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:00.450136 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5p5fm" event={"ID":"e9a5febb-adab-42b2-a740-a057ada6294a","Type":"ContainerStarted","Data":"58bdd0c4855749977cc6e95e56fbbccb339a02b05e77c6b808cd8c04c3a1f602"} Jul 30 19:25:00.451605 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:00.451585 2556 generic.go:356] "Generic (PLEG): container finished" podID="6ed49186-8fa8-4875-8da3-b74f8fc2b712" containerID="386dc1fd25e5c19d800fa3787b85f29379c26e5b18d1b75ea315726627f3618c" exitCode=0 Jul 30 19:25:00.451698 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:00.451644 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r8fml" event={"ID":"6ed49186-8fa8-4875-8da3-b74f8fc2b712","Type":"ContainerDied","Data":"386dc1fd25e5c19d800fa3787b85f29379c26e5b18d1b75ea315726627f3618c"} Jul 30 19:25:00.453172 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:00.453147 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rfjmx" event={"ID":"c058e636-437f-4395-9e4a-760cc5fd051c","Type":"ContainerStarted","Data":"c884acca481df38ec292924277de6a35a5ee911b8687a1623d9bd2dd182484d4"} Jul 30 19:25:00.454595 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:00.454571 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qjcpf" event={"ID":"87b39982-2538-4dcf-baa2-6c522b0c4896","Type":"ContainerStarted","Data":"46460a572b47fafe42c6f40d549eb0599e212e5cd74f73630a44edf428bcecac"} Jul 30 19:25:00.456072 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:00.456048 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ddrgc" event={"ID":"47812e97-b92a-460e-8383-6728edc33f73","Type":"ContainerStarted","Data":"ec5ad4d9b4d53542f0380f217c9cffa570649fc8be5c8ed213cb5d7dfc60e1c0"} Jul 30 19:25:00.462261 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:00.462228 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ttglk" podStartSLOduration=4.2504917429999995 podStartE2EDuration="22.462217479s" totalImagesPullingTime="18.211725736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 19:24:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:25:00.461551672 +0000 UTC m=+22.681110583" watchObservedRunningTime="2026-07-30 19:25:00.462217479 +0000 UTC m=+22.681776443" Jul 30 19:25:00.475146 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:00.475114 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rfjmx" podStartSLOduration=4.24839409 podStartE2EDuration="22.475101549s" totalImagesPullingTime="18.226707459s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 19:24:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:25:00.474719484 +0000 UTC m=+22.694278395" watchObservedRunningTime="2026-07-30 19:25:00.475101549 +0000 UTC m=+22.694660461" Jul 30 19:25:00.511224 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:00.511187 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-ddrgc" podStartSLOduration=4.18123925 podStartE2EDuration="22.51117472s" totalImagesPullingTime="18.32993547s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 19:24:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:25:00.510647045 +0000 UTC m=+22.730205957" watchObservedRunningTime="2026-07-30 19:25:00.51117472 +0000 UTC m=+22.730733629" Jul 30 19:25:00.523018 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:00.522911 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-5p5fm" podStartSLOduration=4.297802454 podStartE2EDuration="22.522896451s" totalImagesPullingTime="18.225093997s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 19:24:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:25:00.522866373 +0000 UTC m=+22.742425283" watchObservedRunningTime="2026-07-30 19:25:00.522896451 +0000 UTC m=+22.742455363" Jul 30 19:25:00.537258 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:00.537227 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qjcpf" podStartSLOduration=4.319813464 podStartE2EDuration="22.537216908s" totalImagesPullingTime="18.217403444s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 19:24:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:25:00.536876597 +0000 UTC m=+22.756435510" watchObservedRunningTime="2026-07-30 19:25:00.537216908 +0000 UTC m=+22.756775819" Jul 30 19:25:01.318700 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:01.318665 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxwhr" Jul 30 19:25:01.319366 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:01.318809 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxwhr" podUID="9547ff50-df97-486d-85eb-c253420d0e73" Jul 30 19:25:01.461722 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:01.461698 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-795qw_4cdd383f-6493-4d83-80fa-f4ce17e8416e/ovn-acl-logging/0.log" Jul 30 19:25:01.462259 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:01.462232 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-795qw" event={"ID":"4cdd383f-6493-4d83-80fa-f4ce17e8416e","Type":"ContainerStarted","Data":"708e1d9b68c5f74b2c6fa3f3f6012febed7d10e50bb291e1043a53c4b160c148"} Jul 30 19:25:01.462456 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:01.462322 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-795qw" event={"ID":"4cdd383f-6493-4d83-80fa-f4ce17e8416e","Type":"ContainerStarted","Data":"e232752cc19070bf3ac5f768a37f9798e781b14c5a6db816ede16eeb74745ae8"} Jul 30 19:25:01.462456 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:01.462374 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-795qw" event={"ID":"4cdd383f-6493-4d83-80fa-f4ce17e8416e","Type":"ContainerStarted","Data":"5aef9f4b123b2b47128379bd7f2d065549aac36e54e186f8c01aba3f98af77a9"} Jul 30 19:25:01.462456 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:01.462439 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-795qw" event={"ID":"4cdd383f-6493-4d83-80fa-f4ce17e8416e","Type":"ContainerStarted","Data":"6cc0e7298111d4f5616e522fb29a718ee5f8433a50732eaaa94d0f62c502bf29"} Jul 30 19:25:01.463912 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:01.463839 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dhdl2" event={"ID":"78e98956-00da-43a5-984d-6b90bbc5f2ca","Type":"ContainerStarted","Data":"6470c4d231a8ca936517379a18bb835673ecbaf3c394662c8a1962fe2b34d1c5"} Jul 30 19:25:01.571086 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:01.571034 2556 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 30 19:25:02.228836 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:02.228713 2556 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-30T19:25:01.571049419Z","UUID":"1b48f850-3e22-45e3-bab9-5b1dc54e8d23","Handler":null,"Name":"","Endpoint":""} Jul 30 19:25:02.230402 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:02.230360 2556 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 30 19:25:02.230536 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:02.230412 2556 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 30 19:25:02.245894 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:02.245865 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ttglk" Jul 30 19:25:02.246485 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:02.246459 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ttglk" Jul 30 19:25:02.264544 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:02.264474 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dhdl2" podStartSLOduration=6.053280997 podStartE2EDuration="24.264461077s" totalImagesPullingTime="18.21118008s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 19:24:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:25:01.499943737 +0000 UTC m=+23.719502651" watchObservedRunningTime="2026-07-30 19:25:02.264461077 +0000 UTC m=+24.484019991" Jul 30 19:25:02.318618 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:02.318587 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-czm5g" Jul 30 19:25:02.318750 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:02.318706 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-czm5g" podUID="c96c7638-b6ce-4a98-a49d-1bf0fb43edf8" Jul 30 19:25:02.319116 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:02.318757 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tpv8" Jul 30 19:25:02.319116 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:02.318842 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6tpv8" podUID="8f32c8f1-7bf6-4b84-8205-555365c55861" Jul 30 19:25:02.468006 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:02.467955 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lp98t" event={"ID":"d1e86eb9-30b6-4e63-8046-3867c587e8f9","Type":"ContainerStarted","Data":"7b4cc24b705d91dee53a04b9a4e42a5919d4d848ba1ff1574777033ca829b552"} Jul 30 19:25:02.468480 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:02.468459 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ttglk" Jul 30 19:25:02.468942 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:02.468923 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ttglk" Jul 30 19:25:03.318094 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:03.318068 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxwhr" Jul 30 19:25:03.318263 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:03.318188 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxwhr" podUID="9547ff50-df97-486d-85eb-c253420d0e73" Jul 30 19:25:04.318970 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:04.318797 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-czm5g" Jul 30 19:25:04.319379 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:04.318852 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tpv8" Jul 30 19:25:04.319379 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:04.319060 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-czm5g" podUID="c96c7638-b6ce-4a98-a49d-1bf0fb43edf8" Jul 30 19:25:04.319379 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:04.319130 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6tpv8" podUID="8f32c8f1-7bf6-4b84-8205-555365c55861" Jul 30 19:25:05.318016 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:05.317951 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxwhr" Jul 30 19:25:05.318148 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:05.318046 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxwhr" podUID="9547ff50-df97-486d-85eb-c253420d0e73" Jul 30 19:25:05.475540 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:05.475509 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lp98t" event={"ID":"d1e86eb9-30b6-4e63-8046-3867c587e8f9","Type":"ContainerStarted","Data":"3827521c5e287a36651a37479c21b0c761599e5ebd3836b9b65814fcb088d61f"} Jul 30 19:25:05.477132 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:05.477107 2556 generic.go:356] "Generic (PLEG): container finished" podID="6ed49186-8fa8-4875-8da3-b74f8fc2b712" containerID="7326c0298c7bc274ad8670d2bdd2ab1f0259d91274b3262f2af450e71c3d55a9" exitCode=0 Jul 30 19:25:05.477250 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:05.477170 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r8fml" event={"ID":"6ed49186-8fa8-4875-8da3-b74f8fc2b712","Type":"ContainerDied","Data":"7326c0298c7bc274ad8670d2bdd2ab1f0259d91274b3262f2af450e71c3d55a9"} Jul 30 19:25:05.480684 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:05.480664 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-795qw_4cdd383f-6493-4d83-80fa-f4ce17e8416e/ovn-acl-logging/0.log" Jul 30 19:25:05.481147 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:05.481124 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-795qw" event={"ID":"4cdd383f-6493-4d83-80fa-f4ce17e8416e","Type":"ContainerStarted","Data":"76f0d2fe2db648b81fd693d419f06a806e063dea3177edd4cd501337221ac6a4"} Jul 30 19:25:05.510287 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:05.510252 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lp98t" podStartSLOduration=4.946650245 podStartE2EDuration="27.510239874s" totalImagesPullingTime="22.563589629s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 19:24:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:25:05.492238563 +0000 UTC m=+27.711797556" watchObservedRunningTime="2026-07-30 19:25:05.510239874 +0000 UTC m=+27.729798785" Jul 30 19:25:06.318350 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:06.318283 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-czm5g" Jul 30 19:25:06.318518 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:06.318292 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tpv8" Jul 30 19:25:06.318518 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:06.318375 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-czm5g" podUID="c96c7638-b6ce-4a98-a49d-1bf0fb43edf8" Jul 30 19:25:06.318518 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:06.318477 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6tpv8" podUID="8f32c8f1-7bf6-4b84-8205-555365c55861" Jul 30 19:25:07.318562 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:07.318311 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxwhr" Jul 30 19:25:07.319140 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:07.318589 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxwhr" podUID="9547ff50-df97-486d-85eb-c253420d0e73" Jul 30 19:25:07.488803 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:07.488776 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-795qw_4cdd383f-6493-4d83-80fa-f4ce17e8416e/ovn-acl-logging/0.log" Jul 30 19:25:07.489135 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:07.489105 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-795qw" event={"ID":"4cdd383f-6493-4d83-80fa-f4ce17e8416e","Type":"ContainerStarted","Data":"ad35d37c52639559eac336e2ef57368d6b7a28b7586fb3d61e5ddb9ea319690f"} Jul 30 19:25:07.489450 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:07.489424 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:25:07.489578 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:07.489558 2556 scope.go:117] "RemoveContainer" containerID="b8fd40b133ae60f84f8be5ed5ed4460c7da14093366f7efdcd1c26cd7c15fc32" Jul 30 19:25:07.491161 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:07.491138 2556 generic.go:356] "Generic (PLEG): container finished" podID="6ed49186-8fa8-4875-8da3-b74f8fc2b712" containerID="2052a5bcc0bc5baffe92f78971cfff48ea9ef573bcd346f2ff3596a13b36b070" exitCode=0 Jul 30 19:25:07.491230 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:07.491171 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r8fml" event={"ID":"6ed49186-8fa8-4875-8da3-b74f8fc2b712","Type":"ContainerDied","Data":"2052a5bcc0bc5baffe92f78971cfff48ea9ef573bcd346f2ff3596a13b36b070"} Jul 30 19:25:07.505574 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:07.505556 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:25:08.319572 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:08.319534 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-czm5g" Jul 30 19:25:08.320019 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:08.319636 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-czm5g" podUID="c96c7638-b6ce-4a98-a49d-1bf0fb43edf8" Jul 30 19:25:08.320019 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:08.319683 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tpv8" Jul 30 19:25:08.320019 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:08.319771 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6tpv8" podUID="8f32c8f1-7bf6-4b84-8205-555365c55861" Jul 30 19:25:08.498626 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:08.498598 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-795qw_4cdd383f-6493-4d83-80fa-f4ce17e8416e/ovn-acl-logging/0.log" Jul 30 19:25:08.499015 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:08.498988 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-795qw" event={"ID":"4cdd383f-6493-4d83-80fa-f4ce17e8416e","Type":"ContainerStarted","Data":"02f18897d4ae9771e2a4dd33dbb78f177ada50a9ba3b29613937687e2a2eaed6"} Jul 30 19:25:08.499494 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:08.499451 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:25:08.499656 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:08.499632 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:25:08.515696 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:08.515677 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:25:08.526876 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:08.526836 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-795qw" podStartSLOduration=11.863326679 podStartE2EDuration="30.526826283s" totalImagesPullingTime="18.663499604s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 19:24:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:25:08.525576001 +0000 UTC m=+30.745134912" watchObservedRunningTime="2026-07-30 19:25:08.526826283 +0000 UTC m=+30.746385193" Jul 30 19:25:08.804109 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:08.803920 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-czm5g"] Jul 30 19:25:08.804222 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:08.804210 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-czm5g" Jul 30 19:25:08.804320 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:08.804301 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-czm5g" podUID="c96c7638-b6ce-4a98-a49d-1bf0fb43edf8" Jul 30 19:25:08.807592 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:08.807564 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6tpv8"] Jul 30 19:25:08.807719 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:08.807702 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tpv8" Jul 30 19:25:08.807823 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:08.807802 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6tpv8" podUID="8f32c8f1-7bf6-4b84-8205-555365c55861" Jul 30 19:25:08.808538 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:08.808420 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vxwhr"] Jul 30 19:25:08.808538 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:08.808525 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxwhr" Jul 30 19:25:08.808671 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:08.808623 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxwhr" podUID="9547ff50-df97-486d-85eb-c253420d0e73" Jul 30 19:25:09.506310 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:09.506280 2556 generic.go:356] "Generic (PLEG): container finished" podID="6ed49186-8fa8-4875-8da3-b74f8fc2b712" containerID="c92204b17a7f1e8c7fb2d4f4b35d20f127c54fb2c6ac78557c9a895dc84c8fb1" exitCode=0 Jul 30 19:25:09.506675 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:09.506415 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r8fml" event={"ID":"6ed49186-8fa8-4875-8da3-b74f8fc2b712","Type":"ContainerDied","Data":"c92204b17a7f1e8c7fb2d4f4b35d20f127c54fb2c6ac78557c9a895dc84c8fb1"} Jul 30 19:25:10.318759 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:10.318720 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxwhr" Jul 30 19:25:10.319454 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:10.319427 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-czm5g" Jul 30 19:25:10.319778 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:10.319745 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxwhr" podUID="9547ff50-df97-486d-85eb-c253420d0e73" Jul 30 19:25:10.320249 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:10.320226 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tpv8" Jul 30 19:25:10.320802 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:10.320772 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-czm5g" podUID="c96c7638-b6ce-4a98-a49d-1bf0fb43edf8" Jul 30 19:25:10.321729 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:10.321655 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6tpv8" podUID="8f32c8f1-7bf6-4b84-8205-555365c55861" Jul 30 19:25:12.031758 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:12.031725 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8f32c8f1-7bf6-4b84-8205-555365c55861-original-pull-secret\") pod \"global-pull-secret-syncer-6tpv8\" (UID: \"8f32c8f1-7bf6-4b84-8205-555365c55861\") " pod="kube-system/global-pull-secret-syncer-6tpv8" Jul 30 19:25:12.032171 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:12.031796 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9547ff50-df97-486d-85eb-c253420d0e73-metrics-certs\") pod \"network-metrics-daemon-vxwhr\" (UID: \"9547ff50-df97-486d-85eb-c253420d0e73\") " pod="openshift-multus/network-metrics-daemon-vxwhr" Jul 30 19:25:12.032171 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:12.031880 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 19:25:12.032171 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:12.031915 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 19:25:12.032171 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:12.031940 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f32c8f1-7bf6-4b84-8205-555365c55861-original-pull-secret podName:8f32c8f1-7bf6-4b84-8205-555365c55861 nodeName:}" failed. No retries permitted until 2026-07-30 19:25:44.031922809 +0000 UTC m=+66.251481700 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8f32c8f1-7bf6-4b84-8205-555365c55861-original-pull-secret") pod "global-pull-secret-syncer-6tpv8" (UID: "8f32c8f1-7bf6-4b84-8205-555365c55861") : object "kube-system"/"original-pull-secret" not registered Jul 30 19:25:12.032171 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:12.031963 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9547ff50-df97-486d-85eb-c253420d0e73-metrics-certs podName:9547ff50-df97-486d-85eb-c253420d0e73 nodeName:}" failed. No retries permitted until 2026-07-30 19:25:44.031946368 +0000 UTC m=+66.251505260 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9547ff50-df97-486d-85eb-c253420d0e73-metrics-certs") pod "network-metrics-daemon-vxwhr" (UID: "9547ff50-df97-486d-85eb-c253420d0e73") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 19:25:12.133135 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:12.133100 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glnkx\" (UniqueName: \"kubernetes.io/projected/c96c7638-b6ce-4a98-a49d-1bf0fb43edf8-kube-api-access-glnkx\") pod \"network-check-target-czm5g\" (UID: \"c96c7638-b6ce-4a98-a49d-1bf0fb43edf8\") " pod="openshift-network-diagnostics/network-check-target-czm5g" Jul 30 19:25:12.133315 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:12.133296 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 19:25:12.133375 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:12.133323 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 19:25:12.133375 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:12.133337 2556 projected.go:196] Error preparing data for projected volume kube-api-access-glnkx for pod openshift-network-diagnostics/network-check-target-czm5g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 19:25:12.133463 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:12.133413 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c96c7638-b6ce-4a98-a49d-1bf0fb43edf8-kube-api-access-glnkx podName:c96c7638-b6ce-4a98-a49d-1bf0fb43edf8 nodeName:}" failed. No retries permitted until 2026-07-30 19:25:44.133378436 +0000 UTC m=+66.352937333 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-glnkx" (UniqueName: "kubernetes.io/projected/c96c7638-b6ce-4a98-a49d-1bf0fb43edf8-kube-api-access-glnkx") pod "network-check-target-czm5g" (UID: "c96c7638-b6ce-4a98-a49d-1bf0fb43edf8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 19:25:12.318791 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:12.318710 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tpv8" Jul 30 19:25:12.318939 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:12.318710 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-czm5g" Jul 30 19:25:12.318939 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:12.318834 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxwhr" Jul 30 19:25:12.318939 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:12.318835 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6tpv8" podUID="8f32c8f1-7bf6-4b84-8205-555365c55861" Jul 30 19:25:12.318939 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:12.318914 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-czm5g" podUID="c96c7638-b6ce-4a98-a49d-1bf0fb43edf8" Jul 30 19:25:12.319131 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:12.319020 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxwhr" podUID="9547ff50-df97-486d-85eb-c253420d0e73" Jul 30 19:25:13.605570 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.605506 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-105.ec2.internal" event="NodeReady" Jul 30 19:25:13.606047 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.605632 2556 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 30 19:25:13.655302 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.655154 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-gng5x"] Jul 30 19:25:13.658128 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.658103 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-896dfc6b7-bck54"] Jul 30 19:25:13.658372 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.658347 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gng5x" Jul 30 19:25:13.659639 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.659623 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-f2czs"] Jul 30 19:25:13.659767 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.659749 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-896dfc6b7-bck54" Jul 30 19:25:13.660974 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.660735 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 30 19:25:13.661074 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.661045 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-8dddr\"" Jul 30 19:25:13.661124 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.661070 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 30 19:25:13.661290 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.661270 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 30 19:25:13.661963 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.661749 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 30 19:25:13.661963 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.661901 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9rs47\"" Jul 30 19:25:13.661963 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.661919 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gng5x"] Jul 30 19:25:13.662238 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.661997 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-f2czs" Jul 30 19:25:13.662442 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.662422 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 30 19:25:13.662442 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.662434 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 30 19:25:13.665306 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.664814 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9htd7\"" Jul 30 19:25:13.665306 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.664829 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 30 19:25:13.665306 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.665029 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 30 19:25:13.665978 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.665924 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-f2czs"] Jul 30 19:25:13.667870 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.667237 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 30 19:25:13.668517 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.668498 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-896dfc6b7-bck54"] Jul 30 19:25:13.845370 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.845325 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/65ac268e-0120-47af-afc7-4f669f07facd-bound-sa-token\") pod \"image-registry-896dfc6b7-bck54\" (UID: \"65ac268e-0120-47af-afc7-4f669f07facd\") " pod="openshift-image-registry/image-registry-896dfc6b7-bck54" Jul 30 19:25:13.845571 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.845378 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/52b30ff5-e40a-45b5-b6bb-80483bb83057-metrics-tls\") pod \"dns-default-f2czs\" (UID: \"52b30ff5-e40a-45b5-b6bb-80483bb83057\") " pod="openshift-dns/dns-default-f2czs" Jul 30 19:25:13.845571 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.845448 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52b30ff5-e40a-45b5-b6bb-80483bb83057-tmp-dir\") pod \"dns-default-f2czs\" (UID: \"52b30ff5-e40a-45b5-b6bb-80483bb83057\") " pod="openshift-dns/dns-default-f2czs" Jul 30 19:25:13.845571 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.845528 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/65ac268e-0120-47af-afc7-4f669f07facd-ca-trust-extracted\") pod \"image-registry-896dfc6b7-bck54\" (UID: \"65ac268e-0120-47af-afc7-4f669f07facd\") " pod="openshift-image-registry/image-registry-896dfc6b7-bck54" Jul 30 19:25:13.845727 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.845582 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/65ac268e-0120-47af-afc7-4f669f07facd-installation-pull-secrets\") pod \"image-registry-896dfc6b7-bck54\" (UID: \"65ac268e-0120-47af-afc7-4f669f07facd\") " pod="openshift-image-registry/image-registry-896dfc6b7-bck54" Jul 30 19:25:13.845727 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.845613 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mr79\" (UniqueName: \"kubernetes.io/projected/65ac268e-0120-47af-afc7-4f669f07facd-kube-api-access-8mr79\") pod \"image-registry-896dfc6b7-bck54\" (UID: \"65ac268e-0120-47af-afc7-4f669f07facd\") " pod="openshift-image-registry/image-registry-896dfc6b7-bck54" Jul 30 19:25:13.845727 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.845641 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/52b30ff5-e40a-45b5-b6bb-80483bb83057-config-volume\") pod \"dns-default-f2czs\" (UID: \"52b30ff5-e40a-45b5-b6bb-80483bb83057\") " pod="openshift-dns/dns-default-f2czs" Jul 30 19:25:13.845727 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.845670 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q45r\" (UniqueName: \"kubernetes.io/projected/52b30ff5-e40a-45b5-b6bb-80483bb83057-kube-api-access-2q45r\") pod \"dns-default-f2czs\" (UID: \"52b30ff5-e40a-45b5-b6bb-80483bb83057\") " pod="openshift-dns/dns-default-f2czs" Jul 30 19:25:13.845727 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.845690 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pxjh\" (UniqueName: \"kubernetes.io/projected/335be97f-3b45-4e7c-b7d9-2a0405943696-kube-api-access-9pxjh\") pod \"ingress-canary-gng5x\" (UID: \"335be97f-3b45-4e7c-b7d9-2a0405943696\") " pod="openshift-ingress-canary/ingress-canary-gng5x" Jul 30 19:25:13.845927 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.845733 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/65ac268e-0120-47af-afc7-4f669f07facd-trusted-ca\") pod \"image-registry-896dfc6b7-bck54\" (UID: \"65ac268e-0120-47af-afc7-4f669f07facd\") " pod="openshift-image-registry/image-registry-896dfc6b7-bck54" Jul 30 19:25:13.845927 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.845769 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/335be97f-3b45-4e7c-b7d9-2a0405943696-cert\") pod \"ingress-canary-gng5x\" (UID: \"335be97f-3b45-4e7c-b7d9-2a0405943696\") " pod="openshift-ingress-canary/ingress-canary-gng5x" Jul 30 19:25:13.845927 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.845810 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/65ac268e-0120-47af-afc7-4f669f07facd-registry-tls\") pod \"image-registry-896dfc6b7-bck54\" (UID: \"65ac268e-0120-47af-afc7-4f669f07facd\") " pod="openshift-image-registry/image-registry-896dfc6b7-bck54" Jul 30 19:25:13.845927 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.845826 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/65ac268e-0120-47af-afc7-4f669f07facd-registry-certificates\") pod \"image-registry-896dfc6b7-bck54\" (UID: \"65ac268e-0120-47af-afc7-4f669f07facd\") " pod="openshift-image-registry/image-registry-896dfc6b7-bck54" Jul 30 19:25:13.845927 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.845845 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/65ac268e-0120-47af-afc7-4f669f07facd-image-registry-private-configuration\") pod \"image-registry-896dfc6b7-bck54\" (UID: \"65ac268e-0120-47af-afc7-4f669f07facd\") " pod="openshift-image-registry/image-registry-896dfc6b7-bck54" Jul 30 19:25:13.946677 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.946608 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/335be97f-3b45-4e7c-b7d9-2a0405943696-cert\") pod \"ingress-canary-gng5x\" (UID: \"335be97f-3b45-4e7c-b7d9-2a0405943696\") " pod="openshift-ingress-canary/ingress-canary-gng5x" Jul 30 19:25:13.946677 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.946660 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/65ac268e-0120-47af-afc7-4f669f07facd-registry-tls\") pod \"image-registry-896dfc6b7-bck54\" (UID: \"65ac268e-0120-47af-afc7-4f669f07facd\") " pod="openshift-image-registry/image-registry-896dfc6b7-bck54" Jul 30 19:25:13.946884 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.946690 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/65ac268e-0120-47af-afc7-4f669f07facd-registry-certificates\") pod \"image-registry-896dfc6b7-bck54\" (UID: \"65ac268e-0120-47af-afc7-4f669f07facd\") " pod="openshift-image-registry/image-registry-896dfc6b7-bck54" Jul 30 19:25:13.946884 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.946723 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/65ac268e-0120-47af-afc7-4f669f07facd-image-registry-private-configuration\") pod \"image-registry-896dfc6b7-bck54\" (UID: \"65ac268e-0120-47af-afc7-4f669f07facd\") " pod="openshift-image-registry/image-registry-896dfc6b7-bck54" Jul 30 19:25:13.946884 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.946757 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/65ac268e-0120-47af-afc7-4f669f07facd-bound-sa-token\") pod \"image-registry-896dfc6b7-bck54\" (UID: \"65ac268e-0120-47af-afc7-4f669f07facd\") " pod="openshift-image-registry/image-registry-896dfc6b7-bck54" Jul 30 19:25:13.946884 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.946775 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/52b30ff5-e40a-45b5-b6bb-80483bb83057-metrics-tls\") pod \"dns-default-f2czs\" (UID: \"52b30ff5-e40a-45b5-b6bb-80483bb83057\") " pod="openshift-dns/dns-default-f2czs" Jul 30 19:25:13.946884 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.946823 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52b30ff5-e40a-45b5-b6bb-80483bb83057-tmp-dir\") pod \"dns-default-f2czs\" (UID: \"52b30ff5-e40a-45b5-b6bb-80483bb83057\") " pod="openshift-dns/dns-default-f2czs" Jul 30 19:25:13.946884 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:13.946842 2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 30 19:25:13.946884 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:13.946866 2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-896dfc6b7-bck54: secret "image-registry-tls" not found Jul 30 19:25:13.947214 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:13.946925 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65ac268e-0120-47af-afc7-4f669f07facd-registry-tls podName:65ac268e-0120-47af-afc7-4f669f07facd nodeName:}" failed. No retries permitted until 2026-07-30 19:25:14.446906049 +0000 UTC m=+36.666464944 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/65ac268e-0120-47af-afc7-4f669f07facd-registry-tls") pod "image-registry-896dfc6b7-bck54" (UID: "65ac268e-0120-47af-afc7-4f669f07facd") : secret "image-registry-tls" not found Jul 30 19:25:13.947214 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:13.946954 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 19:25:13.947214 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:13.947011 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52b30ff5-e40a-45b5-b6bb-80483bb83057-metrics-tls podName:52b30ff5-e40a-45b5-b6bb-80483bb83057 nodeName:}" failed. No retries permitted until 2026-07-30 19:25:14.446991692 +0000 UTC m=+36.666550597 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/52b30ff5-e40a-45b5-b6bb-80483bb83057-metrics-tls") pod "dns-default-f2czs" (UID: "52b30ff5-e40a-45b5-b6bb-80483bb83057") : secret "dns-default-metrics-tls" not found Jul 30 19:25:13.947214 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.946852 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/65ac268e-0120-47af-afc7-4f669f07facd-ca-trust-extracted\") pod \"image-registry-896dfc6b7-bck54\" (UID: \"65ac268e-0120-47af-afc7-4f669f07facd\") " pod="openshift-image-registry/image-registry-896dfc6b7-bck54" Jul 30 19:25:13.947214 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.947199 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/65ac268e-0120-47af-afc7-4f669f07facd-installation-pull-secrets\") pod \"image-registry-896dfc6b7-bck54\" (UID: \"65ac268e-0120-47af-afc7-4f669f07facd\") " pod="openshift-image-registry/image-registry-896dfc6b7-bck54" Jul 30 19:25:13.947512 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.947227 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mr79\" (UniqueName: \"kubernetes.io/projected/65ac268e-0120-47af-afc7-4f669f07facd-kube-api-access-8mr79\") pod \"image-registry-896dfc6b7-bck54\" (UID: \"65ac268e-0120-47af-afc7-4f669f07facd\") " pod="openshift-image-registry/image-registry-896dfc6b7-bck54" Jul 30 19:25:13.947512 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.947252 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/52b30ff5-e40a-45b5-b6bb-80483bb83057-config-volume\") pod \"dns-default-f2czs\" (UID: \"52b30ff5-e40a-45b5-b6bb-80483bb83057\") " pod="openshift-dns/dns-default-f2czs" Jul 30 19:25:13.947512 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.947279 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q45r\" (UniqueName: \"kubernetes.io/projected/52b30ff5-e40a-45b5-b6bb-80483bb83057-kube-api-access-2q45r\") pod \"dns-default-f2czs\" (UID: \"52b30ff5-e40a-45b5-b6bb-80483bb83057\") " pod="openshift-dns/dns-default-f2czs" Jul 30 19:25:13.947512 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.947302 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pxjh\" (UniqueName: \"kubernetes.io/projected/335be97f-3b45-4e7c-b7d9-2a0405943696-kube-api-access-9pxjh\") pod \"ingress-canary-gng5x\" (UID: \"335be97f-3b45-4e7c-b7d9-2a0405943696\") " pod="openshift-ingress-canary/ingress-canary-gng5x" Jul 30 19:25:13.947512 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.947330 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/65ac268e-0120-47af-afc7-4f669f07facd-trusted-ca\") pod \"image-registry-896dfc6b7-bck54\" (UID: \"65ac268e-0120-47af-afc7-4f669f07facd\") " pod="openshift-image-registry/image-registry-896dfc6b7-bck54" Jul 30 19:25:13.947754 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.947591 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52b30ff5-e40a-45b5-b6bb-80483bb83057-tmp-dir\") pod \"dns-default-f2czs\" (UID: \"52b30ff5-e40a-45b5-b6bb-80483bb83057\") " pod="openshift-dns/dns-default-f2czs" Jul 30 19:25:13.948118 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.948088 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/52b30ff5-e40a-45b5-b6bb-80483bb83057-config-volume\") pod \"dns-default-f2czs\" (UID: \"52b30ff5-e40a-45b5-b6bb-80483bb83057\") " pod="openshift-dns/dns-default-f2czs" Jul 30 19:25:13.948443 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.948423 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/65ac268e-0120-47af-afc7-4f669f07facd-ca-trust-extracted\") pod \"image-registry-896dfc6b7-bck54\" (UID: \"65ac268e-0120-47af-afc7-4f669f07facd\") " pod="openshift-image-registry/image-registry-896dfc6b7-bck54" Jul 30 19:25:13.948762 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.948716 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/65ac268e-0120-47af-afc7-4f669f07facd-registry-certificates\") pod \"image-registry-896dfc6b7-bck54\" (UID: \"65ac268e-0120-47af-afc7-4f669f07facd\") " pod="openshift-image-registry/image-registry-896dfc6b7-bck54" Jul 30 19:25:13.948973 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.948926 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/65ac268e-0120-47af-afc7-4f669f07facd-trusted-ca\") pod \"image-registry-896dfc6b7-bck54\" (UID: \"65ac268e-0120-47af-afc7-4f669f07facd\") " pod="openshift-image-registry/image-registry-896dfc6b7-bck54" Jul 30 19:25:13.951596 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.951577 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/65ac268e-0120-47af-afc7-4f669f07facd-installation-pull-secrets\") pod \"image-registry-896dfc6b7-bck54\" (UID: \"65ac268e-0120-47af-afc7-4f669f07facd\") " pod="openshift-image-registry/image-registry-896dfc6b7-bck54" Jul 30 19:25:13.951694 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.951632 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/65ac268e-0120-47af-afc7-4f669f07facd-image-registry-private-configuration\") pod \"image-registry-896dfc6b7-bck54\" (UID: \"65ac268e-0120-47af-afc7-4f669f07facd\") " pod="openshift-image-registry/image-registry-896dfc6b7-bck54" Jul 30 19:25:13.951754 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.951700 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/335be97f-3b45-4e7c-b7d9-2a0405943696-cert\") pod \"ingress-canary-gng5x\" (UID: \"335be97f-3b45-4e7c-b7d9-2a0405943696\") " pod="openshift-ingress-canary/ingress-canary-gng5x" Jul 30 19:25:13.957901 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.957878 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/65ac268e-0120-47af-afc7-4f669f07facd-bound-sa-token\") pod \"image-registry-896dfc6b7-bck54\" (UID: \"65ac268e-0120-47af-afc7-4f669f07facd\") " pod="openshift-image-registry/image-registry-896dfc6b7-bck54" Jul 30 19:25:13.957997 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.957881 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q45r\" (UniqueName: \"kubernetes.io/projected/52b30ff5-e40a-45b5-b6bb-80483bb83057-kube-api-access-2q45r\") pod \"dns-default-f2czs\" (UID: \"52b30ff5-e40a-45b5-b6bb-80483bb83057\") " pod="openshift-dns/dns-default-f2czs" Jul 30 19:25:13.957997 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.957881 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mr79\" (UniqueName: \"kubernetes.io/projected/65ac268e-0120-47af-afc7-4f669f07facd-kube-api-access-8mr79\") pod \"image-registry-896dfc6b7-bck54\" (UID: \"65ac268e-0120-47af-afc7-4f669f07facd\") " pod="openshift-image-registry/image-registry-896dfc6b7-bck54" Jul 30 19:25:13.958997 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.958970 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pxjh\" (UniqueName: \"kubernetes.io/projected/335be97f-3b45-4e7c-b7d9-2a0405943696-kube-api-access-9pxjh\") pod \"ingress-canary-gng5x\" (UID: \"335be97f-3b45-4e7c-b7d9-2a0405943696\") " pod="openshift-ingress-canary/ingress-canary-gng5x" Jul 30 19:25:13.973648 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:13.973620 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gng5x" Jul 30 19:25:14.316500 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:14.316311 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gng5x"] Jul 30 19:25:14.318014 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:14.317986 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxwhr" Jul 30 19:25:14.318014 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:14.318001 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tpv8" Jul 30 19:25:14.318167 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:14.317987 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-czm5g" Jul 30 19:25:14.322216 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:14.321579 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 30 19:25:14.322216 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:14.321653 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 30 19:25:14.322216 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:14.321813 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 30 19:25:14.322216 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:14.322042 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-82x98\"" Jul 30 19:25:14.322216 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:14.322104 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bgfkv\"" Jul 30 19:25:14.322216 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:14.322111 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 30 19:25:14.452468 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:14.452443 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/52b30ff5-e40a-45b5-b6bb-80483bb83057-metrics-tls\") pod \"dns-default-f2czs\" (UID: \"52b30ff5-e40a-45b5-b6bb-80483bb83057\") " pod="openshift-dns/dns-default-f2czs" Jul 30 19:25:14.452577 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:14.452559 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/65ac268e-0120-47af-afc7-4f669f07facd-registry-tls\") pod \"image-registry-896dfc6b7-bck54\" (UID: \"65ac268e-0120-47af-afc7-4f669f07facd\") " pod="openshift-image-registry/image-registry-896dfc6b7-bck54" Jul 30 19:25:14.452628 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:14.452611 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 19:25:14.452665 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:14.452654 2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 30 19:25:14.452703 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:14.452666 2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-896dfc6b7-bck54: secret "image-registry-tls" not found Jul 30 19:25:14.452703 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:14.452698 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52b30ff5-e40a-45b5-b6bb-80483bb83057-metrics-tls podName:52b30ff5-e40a-45b5-b6bb-80483bb83057 nodeName:}" failed. No retries permitted until 2026-07-30 19:25:15.452676334 +0000 UTC m=+37.672235248 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/52b30ff5-e40a-45b5-b6bb-80483bb83057-metrics-tls") pod "dns-default-f2czs" (UID: "52b30ff5-e40a-45b5-b6bb-80483bb83057") : secret "dns-default-metrics-tls" not found Jul 30 19:25:14.452784 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:14.452713 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65ac268e-0120-47af-afc7-4f669f07facd-registry-tls podName:65ac268e-0120-47af-afc7-4f669f07facd nodeName:}" failed. No retries permitted until 2026-07-30 19:25:15.452706655 +0000 UTC m=+37.672265545 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/65ac268e-0120-47af-afc7-4f669f07facd-registry-tls") pod "image-registry-896dfc6b7-bck54" (UID: "65ac268e-0120-47af-afc7-4f669f07facd") : secret "image-registry-tls" not found Jul 30 19:25:14.520505 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:14.520425 2556 generic.go:356] "Generic (PLEG): container finished" podID="6ed49186-8fa8-4875-8da3-b74f8fc2b712" containerID="db9a7a45deb29ec3c5937c15cc51ad355248b8444e93fbd52d1591ad97fca7ec" exitCode=0 Jul 30 19:25:14.520505 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:14.520458 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r8fml" event={"ID":"6ed49186-8fa8-4875-8da3-b74f8fc2b712","Type":"ContainerDied","Data":"db9a7a45deb29ec3c5937c15cc51ad355248b8444e93fbd52d1591ad97fca7ec"} Jul 30 19:25:14.521609 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:14.521592 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gng5x" event={"ID":"335be97f-3b45-4e7c-b7d9-2a0405943696","Type":"ContainerStarted","Data":"29de54f894b45a1c48b0bdd48d0406ccd4c4d608b215b461628e2938d9fd5003"} Jul 30 19:25:15.460454 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:15.460418 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/52b30ff5-e40a-45b5-b6bb-80483bb83057-metrics-tls\") pod \"dns-default-f2czs\" (UID: \"52b30ff5-e40a-45b5-b6bb-80483bb83057\") " pod="openshift-dns/dns-default-f2czs" Jul 30 19:25:15.461194 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:15.460585 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 19:25:15.461194 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:15.460602 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/65ac268e-0120-47af-afc7-4f669f07facd-registry-tls\") pod \"image-registry-896dfc6b7-bck54\" (UID: \"65ac268e-0120-47af-afc7-4f669f07facd\") " pod="openshift-image-registry/image-registry-896dfc6b7-bck54" Jul 30 19:25:15.461194 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:15.460649 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52b30ff5-e40a-45b5-b6bb-80483bb83057-metrics-tls podName:52b30ff5-e40a-45b5-b6bb-80483bb83057 nodeName:}" failed. No retries permitted until 2026-07-30 19:25:17.46063031 +0000 UTC m=+39.680189203 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/52b30ff5-e40a-45b5-b6bb-80483bb83057-metrics-tls") pod "dns-default-f2czs" (UID: "52b30ff5-e40a-45b5-b6bb-80483bb83057") : secret "dns-default-metrics-tls" not found Jul 30 19:25:15.461194 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:15.460707 2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 30 19:25:15.461194 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:15.460722 2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-896dfc6b7-bck54: secret "image-registry-tls" not found Jul 30 19:25:15.461194 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:15.460775 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65ac268e-0120-47af-afc7-4f669f07facd-registry-tls podName:65ac268e-0120-47af-afc7-4f669f07facd nodeName:}" failed. No retries permitted until 2026-07-30 19:25:17.460760201 +0000 UTC m=+39.680319095 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/65ac268e-0120-47af-afc7-4f669f07facd-registry-tls") pod "image-registry-896dfc6b7-bck54" (UID: "65ac268e-0120-47af-afc7-4f669f07facd") : secret "image-registry-tls" not found Jul 30 19:25:15.527601 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:15.527564 2556 generic.go:356] "Generic (PLEG): container finished" podID="6ed49186-8fa8-4875-8da3-b74f8fc2b712" containerID="3b53673e0889c98629b7d58a7894e61e800d4dbfda708f3649a33ca79ed6ec07" exitCode=0 Jul 30 19:25:15.527763 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:15.527617 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r8fml" event={"ID":"6ed49186-8fa8-4875-8da3-b74f8fc2b712","Type":"ContainerDied","Data":"3b53673e0889c98629b7d58a7894e61e800d4dbfda708f3649a33ca79ed6ec07"} Jul 30 19:25:16.534644 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:16.534607 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r8fml" event={"ID":"6ed49186-8fa8-4875-8da3-b74f8fc2b712","Type":"ContainerStarted","Data":"9b2116705bf02dbf8c9509f564089902e3fe32fe9fa2cbf8db59d465d26c79be"} Jul 30 19:25:16.558415 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:16.558338 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-r8fml" podStartSLOduration=9.303873676 podStartE2EDuration="38.558323608s" totalImagesPullingTime="29.254449932s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 19:24:38 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:25:16.556691762 +0000 UTC m=+38.776250676" watchObservedRunningTime="2026-07-30 19:25:16.558323608 +0000 UTC m=+38.777882536" Jul 30 19:25:17.473593 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:17.473321 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/65ac268e-0120-47af-afc7-4f669f07facd-registry-tls\") pod \"image-registry-896dfc6b7-bck54\" (UID: \"65ac268e-0120-47af-afc7-4f669f07facd\") " pod="openshift-image-registry/image-registry-896dfc6b7-bck54" Jul 30 19:25:17.473593 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:17.473559 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/52b30ff5-e40a-45b5-b6bb-80483bb83057-metrics-tls\") pod \"dns-default-f2czs\" (UID: \"52b30ff5-e40a-45b5-b6bb-80483bb83057\") " pod="openshift-dns/dns-default-f2czs" Jul 30 19:25:17.473773 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:17.473487 2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 30 19:25:17.473773 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:17.473618 2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-896dfc6b7-bck54: secret "image-registry-tls" not found Jul 30 19:25:17.473773 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:17.473655 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 19:25:17.473773 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:17.473674 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65ac268e-0120-47af-afc7-4f669f07facd-registry-tls podName:65ac268e-0120-47af-afc7-4f669f07facd nodeName:}" failed. No retries permitted until 2026-07-30 19:25:21.473659451 +0000 UTC m=+43.693218341 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/65ac268e-0120-47af-afc7-4f669f07facd-registry-tls") pod "image-registry-896dfc6b7-bck54" (UID: "65ac268e-0120-47af-afc7-4f669f07facd") : secret "image-registry-tls" not found Jul 30 19:25:17.473773 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:17.473694 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52b30ff5-e40a-45b5-b6bb-80483bb83057-metrics-tls podName:52b30ff5-e40a-45b5-b6bb-80483bb83057 nodeName:}" failed. No retries permitted until 2026-07-30 19:25:21.473681683 +0000 UTC m=+43.693240578 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/52b30ff5-e40a-45b5-b6bb-80483bb83057-metrics-tls") pod "dns-default-f2czs" (UID: "52b30ff5-e40a-45b5-b6bb-80483bb83057") : secret "dns-default-metrics-tls" not found Jul 30 19:25:17.538183 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:17.538149 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gng5x" event={"ID":"335be97f-3b45-4e7c-b7d9-2a0405943696","Type":"ContainerStarted","Data":"f08354e36bcadf682149a14733e958bb064daf1b641a24f63b2308a0a38f32c9"} Jul 30 19:25:17.553249 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:17.553196 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-gng5x" podStartSLOduration=1.823506689 podStartE2EDuration="4.553178769s" totalImagesPullingTime="2.72967208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 19:25:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:25:17.553173959 +0000 UTC m=+39.772732872" watchObservedRunningTime="2026-07-30 19:25:17.553178769 +0000 UTC m=+39.772737661" Jul 30 19:25:20.151602 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:20.151573 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2nhzb"] Jul 30 19:25:20.172262 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:20.172238 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2nhzb"] Jul 30 19:25:20.172400 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:20.172332 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2nhzb" Jul 30 19:25:20.174890 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:20.174872 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 30 19:25:20.174997 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:20.174969 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 30 19:25:20.175192 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:20.175174 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 30 19:25:20.175301 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:20.175261 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 30 19:25:20.175838 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:20.175824 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7gsjq\"" Jul 30 19:25:20.193372 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:20.193350 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6145ba93-9df0-4842-9043-e63f3e67712a-data-volume\") pod \"insights-runtime-extractor-2nhzb\" (UID: \"6145ba93-9df0-4842-9043-e63f3e67712a\") " pod="openshift-insights/insights-runtime-extractor-2nhzb" Jul 30 19:25:20.193462 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:20.193402 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nrnj\" (UniqueName: \"kubernetes.io/projected/6145ba93-9df0-4842-9043-e63f3e67712a-kube-api-access-7nrnj\") pod \"insights-runtime-extractor-2nhzb\" (UID: \"6145ba93-9df0-4842-9043-e63f3e67712a\") " pod="openshift-insights/insights-runtime-extractor-2nhzb" Jul 30 19:25:20.193505 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:20.193472 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6145ba93-9df0-4842-9043-e63f3e67712a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2nhzb\" (UID: \"6145ba93-9df0-4842-9043-e63f3e67712a\") " pod="openshift-insights/insights-runtime-extractor-2nhzb" Jul 30 19:25:20.193505 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:20.193489 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6145ba93-9df0-4842-9043-e63f3e67712a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2nhzb\" (UID: \"6145ba93-9df0-4842-9043-e63f3e67712a\") " pod="openshift-insights/insights-runtime-extractor-2nhzb" Jul 30 19:25:20.193587 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:20.193507 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6145ba93-9df0-4842-9043-e63f3e67712a-crio-socket\") pod \"insights-runtime-extractor-2nhzb\" (UID: \"6145ba93-9df0-4842-9043-e63f3e67712a\") " pod="openshift-insights/insights-runtime-extractor-2nhzb" Jul 30 19:25:20.294119 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:20.294093 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6145ba93-9df0-4842-9043-e63f3e67712a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2nhzb\" (UID: \"6145ba93-9df0-4842-9043-e63f3e67712a\") " pod="openshift-insights/insights-runtime-extractor-2nhzb" Jul 30 19:25:20.294119 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:20.294119 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6145ba93-9df0-4842-9043-e63f3e67712a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2nhzb\" (UID: \"6145ba93-9df0-4842-9043-e63f3e67712a\") " pod="openshift-insights/insights-runtime-extractor-2nhzb" Jul 30 19:25:20.294244 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:20.294139 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6145ba93-9df0-4842-9043-e63f3e67712a-crio-socket\") pod \"insights-runtime-extractor-2nhzb\" (UID: \"6145ba93-9df0-4842-9043-e63f3e67712a\") " pod="openshift-insights/insights-runtime-extractor-2nhzb" Jul 30 19:25:20.294244 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:20.294199 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6145ba93-9df0-4842-9043-e63f3e67712a-data-volume\") pod \"insights-runtime-extractor-2nhzb\" (UID: \"6145ba93-9df0-4842-9043-e63f3e67712a\") " pod="openshift-insights/insights-runtime-extractor-2nhzb" Jul 30 19:25:20.294244 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:20.294202 2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 30 19:25:20.294244 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:20.294232 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nrnj\" (UniqueName: \"kubernetes.io/projected/6145ba93-9df0-4842-9043-e63f3e67712a-kube-api-access-7nrnj\") pod \"insights-runtime-extractor-2nhzb\" (UID: \"6145ba93-9df0-4842-9043-e63f3e67712a\") " pod="openshift-insights/insights-runtime-extractor-2nhzb" Jul 30 19:25:20.294464 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:20.294255 2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 30 19:25:20.294464 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:20.294260 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6145ba93-9df0-4842-9043-e63f3e67712a-kube-rbac-proxy-cm podName:6145ba93-9df0-4842-9043-e63f3e67712a nodeName:}" failed. No retries permitted until 2026-07-30 19:25:20.794245146 +0000 UTC m=+43.013804037 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6145ba93-9df0-4842-9043-e63f3e67712a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2nhzb" (UID: "6145ba93-9df0-4842-9043-e63f3e67712a") : configmap "kube-rbac-proxy" not found Jul 30 19:25:20.294464 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:20.294343 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6145ba93-9df0-4842-9043-e63f3e67712a-insights-runtime-extractor-tls podName:6145ba93-9df0-4842-9043-e63f3e67712a nodeName:}" failed. No retries permitted until 2026-07-30 19:25:20.794324321 +0000 UTC m=+43.013883224 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6145ba93-9df0-4842-9043-e63f3e67712a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2nhzb" (UID: "6145ba93-9df0-4842-9043-e63f3e67712a") : secret "insights-runtime-extractor-tls" not found Jul 30 19:25:20.294464 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:20.294422 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6145ba93-9df0-4842-9043-e63f3e67712a-crio-socket\") pod \"insights-runtime-extractor-2nhzb\" (UID: \"6145ba93-9df0-4842-9043-e63f3e67712a\") " pod="openshift-insights/insights-runtime-extractor-2nhzb" Jul 30 19:25:20.294666 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:20.294594 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6145ba93-9df0-4842-9043-e63f3e67712a-data-volume\") pod \"insights-runtime-extractor-2nhzb\" (UID: \"6145ba93-9df0-4842-9043-e63f3e67712a\") " pod="openshift-insights/insights-runtime-extractor-2nhzb" Jul 30 19:25:20.303971 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:20.303949 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nrnj\" (UniqueName: \"kubernetes.io/projected/6145ba93-9df0-4842-9043-e63f3e67712a-kube-api-access-7nrnj\") pod \"insights-runtime-extractor-2nhzb\" (UID: \"6145ba93-9df0-4842-9043-e63f3e67712a\") " pod="openshift-insights/insights-runtime-extractor-2nhzb" Jul 30 19:25:20.797751 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:20.797717 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6145ba93-9df0-4842-9043-e63f3e67712a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2nhzb\" (UID: \"6145ba93-9df0-4842-9043-e63f3e67712a\") " pod="openshift-insights/insights-runtime-extractor-2nhzb" Jul 30 19:25:20.797751 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:20.797754 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6145ba93-9df0-4842-9043-e63f3e67712a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2nhzb\" (UID: \"6145ba93-9df0-4842-9043-e63f3e67712a\") " pod="openshift-insights/insights-runtime-extractor-2nhzb" Jul 30 19:25:20.797984 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:20.797846 2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 30 19:25:20.797984 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:20.797882 2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 30 19:25:20.797984 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:20.797917 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6145ba93-9df0-4842-9043-e63f3e67712a-kube-rbac-proxy-cm podName:6145ba93-9df0-4842-9043-e63f3e67712a nodeName:}" failed. No retries permitted until 2026-07-30 19:25:21.797899554 +0000 UTC m=+44.017458448 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6145ba93-9df0-4842-9043-e63f3e67712a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2nhzb" (UID: "6145ba93-9df0-4842-9043-e63f3e67712a") : configmap "kube-rbac-proxy" not found Jul 30 19:25:20.797984 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:20.797931 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6145ba93-9df0-4842-9043-e63f3e67712a-insights-runtime-extractor-tls podName:6145ba93-9df0-4842-9043-e63f3e67712a nodeName:}" failed. No retries permitted until 2026-07-30 19:25:21.797925151 +0000 UTC m=+44.017484041 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6145ba93-9df0-4842-9043-e63f3e67712a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2nhzb" (UID: "6145ba93-9df0-4842-9043-e63f3e67712a") : secret "insights-runtime-extractor-tls" not found Jul 30 19:25:21.502133 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:21.502094 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/65ac268e-0120-47af-afc7-4f669f07facd-registry-tls\") pod \"image-registry-896dfc6b7-bck54\" (UID: \"65ac268e-0120-47af-afc7-4f669f07facd\") " pod="openshift-image-registry/image-registry-896dfc6b7-bck54" Jul 30 19:25:21.502133 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:21.502139 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/52b30ff5-e40a-45b5-b6bb-80483bb83057-metrics-tls\") pod \"dns-default-f2czs\" (UID: \"52b30ff5-e40a-45b5-b6bb-80483bb83057\") " pod="openshift-dns/dns-default-f2czs" Jul 30 19:25:21.502589 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:21.502248 2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 30 19:25:21.502589 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:21.502267 2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-896dfc6b7-bck54: secret "image-registry-tls" not found Jul 30 19:25:21.502589 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:21.502279 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 19:25:21.502589 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:21.502324 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65ac268e-0120-47af-afc7-4f669f07facd-registry-tls podName:65ac268e-0120-47af-afc7-4f669f07facd nodeName:}" failed. No retries permitted until 2026-07-30 19:25:29.502306695 +0000 UTC m=+51.721865585 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/65ac268e-0120-47af-afc7-4f669f07facd-registry-tls") pod "image-registry-896dfc6b7-bck54" (UID: "65ac268e-0120-47af-afc7-4f669f07facd") : secret "image-registry-tls" not found Jul 30 19:25:21.502589 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:21.502337 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52b30ff5-e40a-45b5-b6bb-80483bb83057-metrics-tls podName:52b30ff5-e40a-45b5-b6bb-80483bb83057 nodeName:}" failed. No retries permitted until 2026-07-30 19:25:29.502331291 +0000 UTC m=+51.721890181 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/52b30ff5-e40a-45b5-b6bb-80483bb83057-metrics-tls") pod "dns-default-f2czs" (UID: "52b30ff5-e40a-45b5-b6bb-80483bb83057") : secret "dns-default-metrics-tls" not found Jul 30 19:25:21.804937 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:21.804853 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6145ba93-9df0-4842-9043-e63f3e67712a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2nhzb\" (UID: \"6145ba93-9df0-4842-9043-e63f3e67712a\") " pod="openshift-insights/insights-runtime-extractor-2nhzb" Jul 30 19:25:21.804937 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:21.804889 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6145ba93-9df0-4842-9043-e63f3e67712a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2nhzb\" (UID: \"6145ba93-9df0-4842-9043-e63f3e67712a\") " pod="openshift-insights/insights-runtime-extractor-2nhzb" Jul 30 19:25:21.805106 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:21.804982 2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 30 19:25:21.805106 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:21.805021 2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 30 19:25:21.805106 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:21.805053 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6145ba93-9df0-4842-9043-e63f3e67712a-kube-rbac-proxy-cm podName:6145ba93-9df0-4842-9043-e63f3e67712a nodeName:}" failed. No retries permitted until 2026-07-30 19:25:23.80503457 +0000 UTC m=+46.024593464 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6145ba93-9df0-4842-9043-e63f3e67712a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2nhzb" (UID: "6145ba93-9df0-4842-9043-e63f3e67712a") : configmap "kube-rbac-proxy" not found Jul 30 19:25:21.805106 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:21.805069 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6145ba93-9df0-4842-9043-e63f3e67712a-insights-runtime-extractor-tls podName:6145ba93-9df0-4842-9043-e63f3e67712a nodeName:}" failed. No retries permitted until 2026-07-30 19:25:23.805062073 +0000 UTC m=+46.024620964 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6145ba93-9df0-4842-9043-e63f3e67712a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2nhzb" (UID: "6145ba93-9df0-4842-9043-e63f3e67712a") : secret "insights-runtime-extractor-tls" not found Jul 30 19:25:23.024625 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:23.024597 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5p5fm_e9a5febb-adab-42b2-a740-a057ada6294a/dns-node-resolver/0.log" Jul 30 19:25:23.819987 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:23.819950 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6145ba93-9df0-4842-9043-e63f3e67712a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2nhzb\" (UID: \"6145ba93-9df0-4842-9043-e63f3e67712a\") " pod="openshift-insights/insights-runtime-extractor-2nhzb" Jul 30 19:25:23.819987 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:23.819987 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6145ba93-9df0-4842-9043-e63f3e67712a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2nhzb\" (UID: \"6145ba93-9df0-4842-9043-e63f3e67712a\") " pod="openshift-insights/insights-runtime-extractor-2nhzb" Jul 30 19:25:23.820272 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:23.820134 2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 30 19:25:23.820272 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:23.820187 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6145ba93-9df0-4842-9043-e63f3e67712a-insights-runtime-extractor-tls podName:6145ba93-9df0-4842-9043-e63f3e67712a nodeName:}" failed. No retries permitted until 2026-07-30 19:25:27.820174287 +0000 UTC m=+50.039733179 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6145ba93-9df0-4842-9043-e63f3e67712a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2nhzb" (UID: "6145ba93-9df0-4842-9043-e63f3e67712a") : secret "insights-runtime-extractor-tls" not found Jul 30 19:25:23.820683 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:23.820657 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6145ba93-9df0-4842-9043-e63f3e67712a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2nhzb\" (UID: \"6145ba93-9df0-4842-9043-e63f3e67712a\") " pod="openshift-insights/insights-runtime-extractor-2nhzb" Jul 30 19:25:24.225520 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:24.225495 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rfjmx_c058e636-437f-4395-9e4a-760cc5fd051c/node-ca/0.log" Jul 30 19:25:24.824913 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:24.824859 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gng5x_335be97f-3b45-4e7c-b7d9-2a0405943696/serve-healthcheck-canary/0.log" Jul 30 19:25:27.850932 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:27.850899 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6145ba93-9df0-4842-9043-e63f3e67712a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2nhzb\" (UID: \"6145ba93-9df0-4842-9043-e63f3e67712a\") " pod="openshift-insights/insights-runtime-extractor-2nhzb" Jul 30 19:25:27.851295 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:27.851016 2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 30 19:25:27.851295 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:27.851078 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6145ba93-9df0-4842-9043-e63f3e67712a-insights-runtime-extractor-tls podName:6145ba93-9df0-4842-9043-e63f3e67712a nodeName:}" failed. No retries permitted until 2026-07-30 19:25:35.851062703 +0000 UTC m=+58.070621595 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6145ba93-9df0-4842-9043-e63f3e67712a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2nhzb" (UID: "6145ba93-9df0-4842-9043-e63f3e67712a") : secret "insights-runtime-extractor-tls" not found Jul 30 19:25:29.566011 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:29.565982 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/65ac268e-0120-47af-afc7-4f669f07facd-registry-tls\") pod \"image-registry-896dfc6b7-bck54\" (UID: \"65ac268e-0120-47af-afc7-4f669f07facd\") " pod="openshift-image-registry/image-registry-896dfc6b7-bck54" Jul 30 19:25:29.566526 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:29.566024 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/52b30ff5-e40a-45b5-b6bb-80483bb83057-metrics-tls\") pod \"dns-default-f2czs\" (UID: \"52b30ff5-e40a-45b5-b6bb-80483bb83057\") " pod="openshift-dns/dns-default-f2czs" Jul 30 19:25:29.566526 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:29.566124 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 19:25:29.566526 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:29.566132 2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 30 19:25:29.566526 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:29.566148 2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-896dfc6b7-bck54: secret "image-registry-tls" not found Jul 30 19:25:29.566526 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:29.566170 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52b30ff5-e40a-45b5-b6bb-80483bb83057-metrics-tls podName:52b30ff5-e40a-45b5-b6bb-80483bb83057 nodeName:}" failed. No retries permitted until 2026-07-30 19:25:45.566157676 +0000 UTC m=+67.785716567 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/52b30ff5-e40a-45b5-b6bb-80483bb83057-metrics-tls") pod "dns-default-f2czs" (UID: "52b30ff5-e40a-45b5-b6bb-80483bb83057") : secret "dns-default-metrics-tls" not found Jul 30 19:25:29.566526 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:29.566191 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/65ac268e-0120-47af-afc7-4f669f07facd-registry-tls podName:65ac268e-0120-47af-afc7-4f669f07facd nodeName:}" failed. No retries permitted until 2026-07-30 19:25:45.566177471 +0000 UTC m=+67.785736362 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/65ac268e-0120-47af-afc7-4f669f07facd-registry-tls") pod "image-registry-896dfc6b7-bck54" (UID: "65ac268e-0120-47af-afc7-4f669f07facd") : secret "image-registry-tls" not found Jul 30 19:25:35.910743 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:35.910713 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6145ba93-9df0-4842-9043-e63f3e67712a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2nhzb\" (UID: \"6145ba93-9df0-4842-9043-e63f3e67712a\") " pod="openshift-insights/insights-runtime-extractor-2nhzb" Jul 30 19:25:35.912778 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:35.912760 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6145ba93-9df0-4842-9043-e63f3e67712a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2nhzb\" (UID: \"6145ba93-9df0-4842-9043-e63f3e67712a\") " pod="openshift-insights/insights-runtime-extractor-2nhzb" Jul 30 19:25:36.080935 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:36.080903 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2nhzb" Jul 30 19:25:36.196116 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:36.196046 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2nhzb"] Jul 30 19:25:36.198788 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:25:36.198760 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6145ba93_9df0_4842_9043_e63f3e67712a.slice/crio-a264cdc841600da56c15f2c19de09bf5d2b413b3e87cdb012e6070924182f048 WatchSource:0}: Error finding container a264cdc841600da56c15f2c19de09bf5d2b413b3e87cdb012e6070924182f048: Status 404 returned error can't find the container with id a264cdc841600da56c15f2c19de09bf5d2b413b3e87cdb012e6070924182f048 Jul 30 19:25:36.580725 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:36.580644 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2nhzb" event={"ID":"6145ba93-9df0-4842-9043-e63f3e67712a","Type":"ContainerStarted","Data":"767f5c3a49e33fc38c50e385c1e608053a165d65a546ea4b62871f3e67b873a1"} Jul 30 19:25:36.580725 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:36.580699 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2nhzb" event={"ID":"6145ba93-9df0-4842-9043-e63f3e67712a","Type":"ContainerStarted","Data":"a264cdc841600da56c15f2c19de09bf5d2b413b3e87cdb012e6070924182f048"} Jul 30 19:25:37.585604 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:37.585569 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2nhzb" event={"ID":"6145ba93-9df0-4842-9043-e63f3e67712a","Type":"ContainerStarted","Data":"b2482dfa9ad729c26ed3bbde2ab52cbc92bf706393ac60265efc7dc44cd71aed"} Jul 30 19:25:39.592339 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:39.592307 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2nhzb" event={"ID":"6145ba93-9df0-4842-9043-e63f3e67712a","Type":"ContainerStarted","Data":"af3675e30ed67a3a7f534044d41b90b30c1e737ac21c5e9fa69aaf9bfa616324"} Jul 30 19:25:39.613949 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:39.613899 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2nhzb" podStartSLOduration=17.338337792 podStartE2EDuration="19.613886389s" totalImagesPullingTime="2.275548597s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 19:25:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:25:39.613777909 +0000 UTC m=+61.833336836" watchObservedRunningTime="2026-07-30 19:25:39.613886389 +0000 UTC m=+61.833445303" Jul 30 19:25:40.519446 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:40.519367 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-795qw" Jul 30 19:25:44.072801 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:44.072762 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8f32c8f1-7bf6-4b84-8205-555365c55861-original-pull-secret\") pod \"global-pull-secret-syncer-6tpv8\" (UID: \"8f32c8f1-7bf6-4b84-8205-555365c55861\") " pod="kube-system/global-pull-secret-syncer-6tpv8" Jul 30 19:25:44.073236 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:44.072833 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9547ff50-df97-486d-85eb-c253420d0e73-metrics-certs\") pod \"network-metrics-daemon-vxwhr\" (UID: \"9547ff50-df97-486d-85eb-c253420d0e73\") " pod="openshift-multus/network-metrics-daemon-vxwhr" Jul 30 19:25:44.075222 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:44.075201 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 30 19:25:44.075826 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:44.075802 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 30 19:25:44.085714 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:44.085687 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9547ff50-df97-486d-85eb-c253420d0e73-metrics-certs\") pod \"network-metrics-daemon-vxwhr\" (UID: \"9547ff50-df97-486d-85eb-c253420d0e73\") " pod="openshift-multus/network-metrics-daemon-vxwhr" Jul 30 19:25:44.085801 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:44.085789 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8f32c8f1-7bf6-4b84-8205-555365c55861-original-pull-secret\") pod \"global-pull-secret-syncer-6tpv8\" (UID: \"8f32c8f1-7bf6-4b84-8205-555365c55861\") " pod="kube-system/global-pull-secret-syncer-6tpv8" Jul 30 19:25:44.173842 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:44.173816 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glnkx\" (UniqueName: \"kubernetes.io/projected/c96c7638-b6ce-4a98-a49d-1bf0fb43edf8-kube-api-access-glnkx\") pod \"network-check-target-czm5g\" (UID: \"c96c7638-b6ce-4a98-a49d-1bf0fb43edf8\") " pod="openshift-network-diagnostics/network-check-target-czm5g" Jul 30 19:25:44.176285 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:44.176270 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 30 19:25:44.186214 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:44.186195 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 30 19:25:44.197050 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:44.197027 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glnkx\" (UniqueName: \"kubernetes.io/projected/c96c7638-b6ce-4a98-a49d-1bf0fb43edf8-kube-api-access-glnkx\") pod \"network-check-target-czm5g\" (UID: \"c96c7638-b6ce-4a98-a49d-1bf0fb43edf8\") " pod="openshift-network-diagnostics/network-check-target-czm5g" Jul 30 19:25:44.334762 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:44.334699 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6tpv8" Jul 30 19:25:44.343600 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:44.343581 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-82x98\"" Jul 30 19:25:44.347689 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:44.347668 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bgfkv\"" Jul 30 19:25:44.351445 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:44.351426 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-czm5g" Jul 30 19:25:44.356262 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:44.356243 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxwhr" Jul 30 19:25:44.491111 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:44.491082 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6tpv8"] Jul 30 19:25:44.493972 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:25:44.493946 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f32c8f1_7bf6_4b84_8205_555365c55861.slice/crio-68fda1e9ae7e569eb9b4458302b01a6f2c27a48dc285d2b6c68be8d3edc9b675 WatchSource:0}: Error finding container 68fda1e9ae7e569eb9b4458302b01a6f2c27a48dc285d2b6c68be8d3edc9b675: Status 404 returned error can't find the container with id 68fda1e9ae7e569eb9b4458302b01a6f2c27a48dc285d2b6c68be8d3edc9b675 Jul 30 19:25:44.507064 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:44.507042 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-czm5g"] Jul 30 19:25:44.509943 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:25:44.509916 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc96c7638_b6ce_4a98_a49d_1bf0fb43edf8.slice/crio-62ea526c66a767ea6d5aa85ef8c9791223a8345851100db5aceec90ba0f6cd92 WatchSource:0}: Error finding container 62ea526c66a767ea6d5aa85ef8c9791223a8345851100db5aceec90ba0f6cd92: Status 404 returned error can't find the container with id 62ea526c66a767ea6d5aa85ef8c9791223a8345851100db5aceec90ba0f6cd92 Jul 30 19:25:44.522903 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:44.522882 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vxwhr"] Jul 30 19:25:44.525414 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:25:44.525375 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9547ff50_df97_486d_85eb_c253420d0e73.slice/crio-bc7855aa7d7a5f367248bfc48e6bf2edd86ee32176e01c2683f6039f45c54d9c WatchSource:0}: Error finding container bc7855aa7d7a5f367248bfc48e6bf2edd86ee32176e01c2683f6039f45c54d9c: Status 404 returned error can't find the container with id bc7855aa7d7a5f367248bfc48e6bf2edd86ee32176e01c2683f6039f45c54d9c Jul 30 19:25:44.604030 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:44.603966 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vxwhr" event={"ID":"9547ff50-df97-486d-85eb-c253420d0e73","Type":"ContainerStarted","Data":"bc7855aa7d7a5f367248bfc48e6bf2edd86ee32176e01c2683f6039f45c54d9c"} Jul 30 19:25:44.605003 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:44.604979 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-czm5g" event={"ID":"c96c7638-b6ce-4a98-a49d-1bf0fb43edf8","Type":"ContainerStarted","Data":"62ea526c66a767ea6d5aa85ef8c9791223a8345851100db5aceec90ba0f6cd92"} Jul 30 19:25:44.605944 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:44.605921 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6tpv8" event={"ID":"8f32c8f1-7bf6-4b84-8205-555365c55861","Type":"ContainerStarted","Data":"68fda1e9ae7e569eb9b4458302b01a6f2c27a48dc285d2b6c68be8d3edc9b675"} Jul 30 19:25:45.005356 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.005328 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-97f775b68-lhfdc"] Jul 30 19:25:45.007823 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.007799 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-97f775b68-lhfdc" Jul 30 19:25:45.010476 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.010455 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 30 19:25:45.011114 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.011101 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 30 19:25:45.011260 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.011239 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 30 19:25:45.012253 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.012237 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 30 19:25:45.018648 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.018628 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-97f775b68-lhfdc"] Jul 30 19:25:45.049242 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.049220 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-896dfc6b7-bck54"] Jul 30 19:25:45.049454 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:45.049435 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-896dfc6b7-bck54" podUID="65ac268e-0120-47af-afc7-4f669f07facd" Jul 30 19:25:45.080763 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.080594 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c642\" (UniqueName: \"kubernetes.io/projected/7f32d523-d4ca-4137-a78b-283cc30e0b2b-kube-api-access-6c642\") pod \"klusterlet-addon-workmgr-97f775b68-lhfdc\" (UID: \"7f32d523-d4ca-4137-a78b-283cc30e0b2b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-97f775b68-lhfdc" Jul 30 19:25:45.080763 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.080634 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f32d523-d4ca-4137-a78b-283cc30e0b2b-tmp\") pod \"klusterlet-addon-workmgr-97f775b68-lhfdc\" (UID: \"7f32d523-d4ca-4137-a78b-283cc30e0b2b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-97f775b68-lhfdc" Jul 30 19:25:45.080763 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.080679 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7f32d523-d4ca-4137-a78b-283cc30e0b2b-klusterlet-config\") pod \"klusterlet-addon-workmgr-97f775b68-lhfdc\" (UID: \"7f32d523-d4ca-4137-a78b-283cc30e0b2b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-97f775b68-lhfdc" Jul 30 19:25:45.108623 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.107656 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-77c974ffcf-zvqfw"] Jul 30 19:25:45.109928 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.109904 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-bbb6865cb-vjk5x"] Jul 30 19:25:45.110363 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.110342 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-77c974ffcf-zvqfw" Jul 30 19:25:45.111703 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.111685 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bbb6865cb-vjk5x" Jul 30 19:25:45.113675 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.112997 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 30 19:25:45.113675 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.113234 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 30 19:25:45.113675 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.113521 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-z266m\"" Jul 30 19:25:45.118604 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.118587 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-77c974ffcf-zvqfw"] Jul 30 19:25:45.121534 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.121514 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-bbb6865cb-vjk5x"] Jul 30 19:25:45.183408 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.182067 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c642\" (UniqueName: \"kubernetes.io/projected/7f32d523-d4ca-4137-a78b-283cc30e0b2b-kube-api-access-6c642\") pod \"klusterlet-addon-workmgr-97f775b68-lhfdc\" (UID: \"7f32d523-d4ca-4137-a78b-283cc30e0b2b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-97f775b68-lhfdc" Jul 30 19:25:45.183408 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.182113 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f32d523-d4ca-4137-a78b-283cc30e0b2b-tmp\") pod \"klusterlet-addon-workmgr-97f775b68-lhfdc\" (UID: \"7f32d523-d4ca-4137-a78b-283cc30e0b2b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-97f775b68-lhfdc" Jul 30 19:25:45.183408 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.182148 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c652215e-79e4-4fbc-a464-c6009405517c-registry-certificates\") pod \"image-registry-bbb6865cb-vjk5x\" (UID: \"c652215e-79e4-4fbc-a464-c6009405517c\") " pod="openshift-image-registry/image-registry-bbb6865cb-vjk5x" Jul 30 19:25:45.183408 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.182173 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7vk7\" (UniqueName: \"kubernetes.io/projected/c652215e-79e4-4fbc-a464-c6009405517c-kube-api-access-s7vk7\") pod \"image-registry-bbb6865cb-vjk5x\" (UID: \"c652215e-79e4-4fbc-a464-c6009405517c\") " pod="openshift-image-registry/image-registry-bbb6865cb-vjk5x" Jul 30 19:25:45.183408 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.182212 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c652215e-79e4-4fbc-a464-c6009405517c-trusted-ca\") pod \"image-registry-bbb6865cb-vjk5x\" (UID: \"c652215e-79e4-4fbc-a464-c6009405517c\") " pod="openshift-image-registry/image-registry-bbb6865cb-vjk5x" Jul 30 19:25:45.183408 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.182240 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c652215e-79e4-4fbc-a464-c6009405517c-image-registry-private-configuration\") pod \"image-registry-bbb6865cb-vjk5x\" (UID: \"c652215e-79e4-4fbc-a464-c6009405517c\") " pod="openshift-image-registry/image-registry-bbb6865cb-vjk5x" Jul 30 19:25:45.183408 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.182269 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np8fk\" (UniqueName: \"kubernetes.io/projected/68ac40f9-2822-4b96-8fb5-7d37d192bb97-kube-api-access-np8fk\") pod \"downloads-77c974ffcf-zvqfw\" (UID: \"68ac40f9-2822-4b96-8fb5-7d37d192bb97\") " pod="openshift-console/downloads-77c974ffcf-zvqfw" Jul 30 19:25:45.183408 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.182297 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c652215e-79e4-4fbc-a464-c6009405517c-ca-trust-extracted\") pod \"image-registry-bbb6865cb-vjk5x\" (UID: \"c652215e-79e4-4fbc-a464-c6009405517c\") " pod="openshift-image-registry/image-registry-bbb6865cb-vjk5x" Jul 30 19:25:45.183408 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.182338 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c652215e-79e4-4fbc-a464-c6009405517c-bound-sa-token\") pod \"image-registry-bbb6865cb-vjk5x\" (UID: \"c652215e-79e4-4fbc-a464-c6009405517c\") " pod="openshift-image-registry/image-registry-bbb6865cb-vjk5x" Jul 30 19:25:45.183408 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.182406 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7f32d523-d4ca-4137-a78b-283cc30e0b2b-klusterlet-config\") pod \"klusterlet-addon-workmgr-97f775b68-lhfdc\" (UID: \"7f32d523-d4ca-4137-a78b-283cc30e0b2b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-97f775b68-lhfdc" Jul 30 19:25:45.183408 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.182451 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c652215e-79e4-4fbc-a464-c6009405517c-installation-pull-secrets\") pod \"image-registry-bbb6865cb-vjk5x\" (UID: \"c652215e-79e4-4fbc-a464-c6009405517c\") " pod="openshift-image-registry/image-registry-bbb6865cb-vjk5x" Jul 30 19:25:45.183408 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.182480 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c652215e-79e4-4fbc-a464-c6009405517c-registry-tls\") pod \"image-registry-bbb6865cb-vjk5x\" (UID: \"c652215e-79e4-4fbc-a464-c6009405517c\") " pod="openshift-image-registry/image-registry-bbb6865cb-vjk5x" Jul 30 19:25:45.183408 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.183116 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f32d523-d4ca-4137-a78b-283cc30e0b2b-tmp\") pod \"klusterlet-addon-workmgr-97f775b68-lhfdc\" (UID: \"7f32d523-d4ca-4137-a78b-283cc30e0b2b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-97f775b68-lhfdc" Jul 30 19:25:45.187205 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.187121 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7f32d523-d4ca-4137-a78b-283cc30e0b2b-klusterlet-config\") pod \"klusterlet-addon-workmgr-97f775b68-lhfdc\" (UID: \"7f32d523-d4ca-4137-a78b-283cc30e0b2b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-97f775b68-lhfdc" Jul 30 19:25:45.196652 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.196612 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c642\" (UniqueName: \"kubernetes.io/projected/7f32d523-d4ca-4137-a78b-283cc30e0b2b-kube-api-access-6c642\") pod \"klusterlet-addon-workmgr-97f775b68-lhfdc\" (UID: \"7f32d523-d4ca-4137-a78b-283cc30e0b2b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-97f775b68-lhfdc" Jul 30 19:25:45.282855 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.282776 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c652215e-79e4-4fbc-a464-c6009405517c-installation-pull-secrets\") pod \"image-registry-bbb6865cb-vjk5x\" (UID: \"c652215e-79e4-4fbc-a464-c6009405517c\") " pod="openshift-image-registry/image-registry-bbb6865cb-vjk5x" Jul 30 19:25:45.282855 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.282828 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c652215e-79e4-4fbc-a464-c6009405517c-registry-tls\") pod \"image-registry-bbb6865cb-vjk5x\" (UID: \"c652215e-79e4-4fbc-a464-c6009405517c\") " pod="openshift-image-registry/image-registry-bbb6865cb-vjk5x" Jul 30 19:25:45.283069 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.282884 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c652215e-79e4-4fbc-a464-c6009405517c-registry-certificates\") pod \"image-registry-bbb6865cb-vjk5x\" (UID: \"c652215e-79e4-4fbc-a464-c6009405517c\") " pod="openshift-image-registry/image-registry-bbb6865cb-vjk5x" Jul 30 19:25:45.283069 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.282911 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7vk7\" (UniqueName: \"kubernetes.io/projected/c652215e-79e4-4fbc-a464-c6009405517c-kube-api-access-s7vk7\") pod \"image-registry-bbb6865cb-vjk5x\" (UID: \"c652215e-79e4-4fbc-a464-c6009405517c\") " pod="openshift-image-registry/image-registry-bbb6865cb-vjk5x" Jul 30 19:25:45.283069 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.282951 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c652215e-79e4-4fbc-a464-c6009405517c-trusted-ca\") pod \"image-registry-bbb6865cb-vjk5x\" (UID: \"c652215e-79e4-4fbc-a464-c6009405517c\") " pod="openshift-image-registry/image-registry-bbb6865cb-vjk5x" Jul 30 19:25:45.283069 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.282977 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c652215e-79e4-4fbc-a464-c6009405517c-image-registry-private-configuration\") pod \"image-registry-bbb6865cb-vjk5x\" (UID: \"c652215e-79e4-4fbc-a464-c6009405517c\") " pod="openshift-image-registry/image-registry-bbb6865cb-vjk5x" Jul 30 19:25:45.283069 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.283007 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np8fk\" (UniqueName: \"kubernetes.io/projected/68ac40f9-2822-4b96-8fb5-7d37d192bb97-kube-api-access-np8fk\") pod \"downloads-77c974ffcf-zvqfw\" (UID: \"68ac40f9-2822-4b96-8fb5-7d37d192bb97\") " pod="openshift-console/downloads-77c974ffcf-zvqfw" Jul 30 19:25:45.283069 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.283048 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c652215e-79e4-4fbc-a464-c6009405517c-ca-trust-extracted\") pod \"image-registry-bbb6865cb-vjk5x\" (UID: \"c652215e-79e4-4fbc-a464-c6009405517c\") " pod="openshift-image-registry/image-registry-bbb6865cb-vjk5x" Jul 30 19:25:45.283356 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.283085 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c652215e-79e4-4fbc-a464-c6009405517c-bound-sa-token\") pod \"image-registry-bbb6865cb-vjk5x\" (UID: \"c652215e-79e4-4fbc-a464-c6009405517c\") " pod="openshift-image-registry/image-registry-bbb6865cb-vjk5x" Jul 30 19:25:45.283948 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.283913 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c652215e-79e4-4fbc-a464-c6009405517c-ca-trust-extracted\") pod \"image-registry-bbb6865cb-vjk5x\" (UID: \"c652215e-79e4-4fbc-a464-c6009405517c\") " pod="openshift-image-registry/image-registry-bbb6865cb-vjk5x" Jul 30 19:25:45.284132 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.284112 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c652215e-79e4-4fbc-a464-c6009405517c-registry-certificates\") pod \"image-registry-bbb6865cb-vjk5x\" (UID: \"c652215e-79e4-4fbc-a464-c6009405517c\") " pod="openshift-image-registry/image-registry-bbb6865cb-vjk5x" Jul 30 19:25:45.284728 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.284661 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c652215e-79e4-4fbc-a464-c6009405517c-trusted-ca\") pod \"image-registry-bbb6865cb-vjk5x\" (UID: \"c652215e-79e4-4fbc-a464-c6009405517c\") " pod="openshift-image-registry/image-registry-bbb6865cb-vjk5x" Jul 30 19:25:45.286448 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.286406 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c652215e-79e4-4fbc-a464-c6009405517c-image-registry-private-configuration\") pod \"image-registry-bbb6865cb-vjk5x\" (UID: \"c652215e-79e4-4fbc-a464-c6009405517c\") " pod="openshift-image-registry/image-registry-bbb6865cb-vjk5x" Jul 30 19:25:45.289226 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.288970 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c652215e-79e4-4fbc-a464-c6009405517c-registry-tls\") pod \"image-registry-bbb6865cb-vjk5x\" (UID: \"c652215e-79e4-4fbc-a464-c6009405517c\") " pod="openshift-image-registry/image-registry-bbb6865cb-vjk5x" Jul 30 19:25:45.290368 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.290326 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c652215e-79e4-4fbc-a464-c6009405517c-installation-pull-secrets\") pod \"image-registry-bbb6865cb-vjk5x\" (UID: \"c652215e-79e4-4fbc-a464-c6009405517c\") " pod="openshift-image-registry/image-registry-bbb6865cb-vjk5x" Jul 30 19:25:45.291499 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.291478 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c652215e-79e4-4fbc-a464-c6009405517c-bound-sa-token\") pod \"image-registry-bbb6865cb-vjk5x\" (UID: \"c652215e-79e4-4fbc-a464-c6009405517c\") " pod="openshift-image-registry/image-registry-bbb6865cb-vjk5x" Jul 30 19:25:45.293106 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.293084 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-np8fk\" (UniqueName: \"kubernetes.io/projected/68ac40f9-2822-4b96-8fb5-7d37d192bb97-kube-api-access-np8fk\") pod \"downloads-77c974ffcf-zvqfw\" (UID: \"68ac40f9-2822-4b96-8fb5-7d37d192bb97\") " pod="openshift-console/downloads-77c974ffcf-zvqfw" Jul 30 19:25:45.297757 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.297737 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7vk7\" (UniqueName: \"kubernetes.io/projected/c652215e-79e4-4fbc-a464-c6009405517c-kube-api-access-s7vk7\") pod \"image-registry-bbb6865cb-vjk5x\" (UID: \"c652215e-79e4-4fbc-a464-c6009405517c\") " pod="openshift-image-registry/image-registry-bbb6865cb-vjk5x" Jul 30 19:25:45.317330 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.317130 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-97f775b68-lhfdc" Jul 30 19:25:45.425761 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.425715 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-77c974ffcf-zvqfw" Jul 30 19:25:45.434199 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.434140 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9rs47\"" Jul 30 19:25:45.442636 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.442436 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bbb6865cb-vjk5x" Jul 30 19:25:45.469555 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.469440 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-97f775b68-lhfdc"] Jul 30 19:25:45.586069 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.585991 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/65ac268e-0120-47af-afc7-4f669f07facd-registry-tls\") pod \"image-registry-896dfc6b7-bck54\" (UID: \"65ac268e-0120-47af-afc7-4f669f07facd\") " pod="openshift-image-registry/image-registry-896dfc6b7-bck54" Jul 30 19:25:45.586668 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.586078 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/52b30ff5-e40a-45b5-b6bb-80483bb83057-metrics-tls\") pod \"dns-default-f2czs\" (UID: \"52b30ff5-e40a-45b5-b6bb-80483bb83057\") " pod="openshift-dns/dns-default-f2czs" Jul 30 19:25:45.589313 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.589277 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/52b30ff5-e40a-45b5-b6bb-80483bb83057-metrics-tls\") pod \"dns-default-f2czs\" (UID: \"52b30ff5-e40a-45b5-b6bb-80483bb83057\") " pod="openshift-dns/dns-default-f2czs" Jul 30 19:25:45.589438 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.589283 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/65ac268e-0120-47af-afc7-4f669f07facd-registry-tls\") pod \"image-registry-896dfc6b7-bck54\" (UID: \"65ac268e-0120-47af-afc7-4f669f07facd\") " pod="openshift-image-registry/image-registry-896dfc6b7-bck54" Jul 30 19:25:45.610108 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.609704 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-896dfc6b7-bck54" Jul 30 19:25:45.621637 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.621613 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-896dfc6b7-bck54" Jul 30 19:25:45.687740 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.687665 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/65ac268e-0120-47af-afc7-4f669f07facd-registry-certificates\") pod \"65ac268e-0120-47af-afc7-4f669f07facd\" (UID: \"65ac268e-0120-47af-afc7-4f669f07facd\") " Jul 30 19:25:45.687740 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.687712 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/65ac268e-0120-47af-afc7-4f669f07facd-bound-sa-token\") pod \"65ac268e-0120-47af-afc7-4f669f07facd\" (UID: \"65ac268e-0120-47af-afc7-4f669f07facd\") " Jul 30 19:25:45.687740 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.687744 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/65ac268e-0120-47af-afc7-4f669f07facd-image-registry-private-configuration\") pod \"65ac268e-0120-47af-afc7-4f669f07facd\" (UID: \"65ac268e-0120-47af-afc7-4f669f07facd\") " Jul 30 19:25:45.688006 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.687787 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8mr79\" (UniqueName: \"kubernetes.io/projected/65ac268e-0120-47af-afc7-4f669f07facd-kube-api-access-8mr79\") pod \"65ac268e-0120-47af-afc7-4f669f07facd\" (UID: \"65ac268e-0120-47af-afc7-4f669f07facd\") " Jul 30 19:25:45.688006 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.687816 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/65ac268e-0120-47af-afc7-4f669f07facd-registry-tls\") pod \"65ac268e-0120-47af-afc7-4f669f07facd\" (UID: \"65ac268e-0120-47af-afc7-4f669f07facd\") " Jul 30 19:25:45.688125 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.688034 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65ac268e-0120-47af-afc7-4f669f07facd-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "65ac268e-0120-47af-afc7-4f669f07facd" (UID: "65ac268e-0120-47af-afc7-4f669f07facd"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 19:25:45.688125 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.688042 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/65ac268e-0120-47af-afc7-4f669f07facd-ca-trust-extracted\") pod \"65ac268e-0120-47af-afc7-4f669f07facd\" (UID: \"65ac268e-0120-47af-afc7-4f669f07facd\") " Jul 30 19:25:45.688223 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.688161 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/65ac268e-0120-47af-afc7-4f669f07facd-installation-pull-secrets\") pod \"65ac268e-0120-47af-afc7-4f669f07facd\" (UID: \"65ac268e-0120-47af-afc7-4f669f07facd\") " Jul 30 19:25:45.688275 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.688246 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65ac268e-0120-47af-afc7-4f669f07facd-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "65ac268e-0120-47af-afc7-4f669f07facd" (UID: "65ac268e-0120-47af-afc7-4f669f07facd"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 19:25:45.688275 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.688252 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/65ac268e-0120-47af-afc7-4f669f07facd-trusted-ca\") pod \"65ac268e-0120-47af-afc7-4f669f07facd\" (UID: \"65ac268e-0120-47af-afc7-4f669f07facd\") " Jul 30 19:25:45.688482 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.688458 2556 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/65ac268e-0120-47af-afc7-4f669f07facd-ca-trust-extracted\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:25:45.688482 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.688484 2556 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/65ac268e-0120-47af-afc7-4f669f07facd-registry-certificates\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:25:45.688723 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.688697 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65ac268e-0120-47af-afc7-4f669f07facd-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "65ac268e-0120-47af-afc7-4f669f07facd" (UID: "65ac268e-0120-47af-afc7-4f669f07facd"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 19:25:45.690553 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.690525 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65ac268e-0120-47af-afc7-4f669f07facd-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "65ac268e-0120-47af-afc7-4f669f07facd" (UID: "65ac268e-0120-47af-afc7-4f669f07facd"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 19:25:45.690871 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.690848 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65ac268e-0120-47af-afc7-4f669f07facd-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "65ac268e-0120-47af-afc7-4f669f07facd" (UID: "65ac268e-0120-47af-afc7-4f669f07facd"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 19:25:45.690984 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.690955 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65ac268e-0120-47af-afc7-4f669f07facd-kube-api-access-8mr79" (OuterVolumeSpecName: "kube-api-access-8mr79") pod "65ac268e-0120-47af-afc7-4f669f07facd" (UID: "65ac268e-0120-47af-afc7-4f669f07facd"). InnerVolumeSpecName "kube-api-access-8mr79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 19:25:45.692512 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.692487 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65ac268e-0120-47af-afc7-4f669f07facd-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "65ac268e-0120-47af-afc7-4f669f07facd" (UID: "65ac268e-0120-47af-afc7-4f669f07facd"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 19:25:45.692664 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.692640 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65ac268e-0120-47af-afc7-4f669f07facd-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "65ac268e-0120-47af-afc7-4f669f07facd" (UID: "65ac268e-0120-47af-afc7-4f669f07facd"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 19:25:45.789534 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.789475 2556 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/65ac268e-0120-47af-afc7-4f669f07facd-installation-pull-secrets\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:25:45.789534 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.789507 2556 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/65ac268e-0120-47af-afc7-4f669f07facd-trusted-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:25:45.789534 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.789521 2556 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/65ac268e-0120-47af-afc7-4f669f07facd-bound-sa-token\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:25:45.789534 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.789534 2556 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/65ac268e-0120-47af-afc7-4f669f07facd-image-registry-private-configuration\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:25:45.789826 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.789546 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mr79\" (UniqueName: \"kubernetes.io/projected/65ac268e-0120-47af-afc7-4f669f07facd-kube-api-access-8mr79\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:25:45.789826 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.789558 2556 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/65ac268e-0120-47af-afc7-4f669f07facd-registry-tls\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:25:45.790669 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.790595 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9htd7\"" Jul 30 19:25:45.799322 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.799129 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-f2czs" Jul 30 19:25:45.904341 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.904187 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-77c974ffcf-zvqfw"] Jul 30 19:25:45.908267 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:25:45.908233 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68ac40f9_2822_4b96_8fb5_7d37d192bb97.slice/crio-dfac691d96cbd43bffcd3847e68a44ce3c8ce165e5cee86dc8123c37fe6b3339 WatchSource:0}: Error finding container dfac691d96cbd43bffcd3847e68a44ce3c8ce165e5cee86dc8123c37fe6b3339: Status 404 returned error can't find the container with id dfac691d96cbd43bffcd3847e68a44ce3c8ce165e5cee86dc8123c37fe6b3339 Jul 30 19:25:45.935484 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.935019 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-bbb6865cb-vjk5x"] Jul 30 19:25:45.971642 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:45.971610 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-f2czs"] Jul 30 19:25:46.085550 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:25:46.085507 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc652215e_79e4_4fbc_a464_c6009405517c.slice/crio-a74e430a283b79c87389ba2937dee784c874758f698f86cb5be9a2dc7d5f16e1 WatchSource:0}: Error finding container a74e430a283b79c87389ba2937dee784c874758f698f86cb5be9a2dc7d5f16e1: Status 404 returned error can't find the container with id a74e430a283b79c87389ba2937dee784c874758f698f86cb5be9a2dc7d5f16e1 Jul 30 19:25:46.619441 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:46.619177 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vxwhr" event={"ID":"9547ff50-df97-486d-85eb-c253420d0e73","Type":"ContainerStarted","Data":"29eee248b4d7e986c4436140de48e82b95c603cf136be257b83588ac8668522a"} Jul 30 19:25:46.619441 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:46.619232 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vxwhr" event={"ID":"9547ff50-df97-486d-85eb-c253420d0e73","Type":"ContainerStarted","Data":"4a1a0e3ab531589fd826b332b251187da780dd6a0453a2d657b8f9532c2b1f9d"} Jul 30 19:25:46.620732 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:46.620690 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-f2czs" event={"ID":"52b30ff5-e40a-45b5-b6bb-80483bb83057","Type":"ContainerStarted","Data":"868696db48c498c66fec7d4af17902190eda7649f3b5a75d3e3679d899059dc9"} Jul 30 19:25:46.622904 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:46.622877 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bbb6865cb-vjk5x" event={"ID":"c652215e-79e4-4fbc-a464-c6009405517c","Type":"ContainerStarted","Data":"8f29040b79a12166d9a0bde32905be6d7ee57b7979f376c468c06854b7f05b68"} Jul 30 19:25:46.623017 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:46.622922 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bbb6865cb-vjk5x" event={"ID":"c652215e-79e4-4fbc-a464-c6009405517c","Type":"ContainerStarted","Data":"a74e430a283b79c87389ba2937dee784c874758f698f86cb5be9a2dc7d5f16e1"} Jul 30 19:25:46.623440 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:46.623421 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-bbb6865cb-vjk5x" Jul 30 19:25:46.625121 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:46.625100 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-97f775b68-lhfdc" event={"ID":"7f32d523-d4ca-4137-a78b-283cc30e0b2b","Type":"ContainerStarted","Data":"fa3a55d05edecc18f1ecd91d3a898e857742f17fd8174528535f15186c974c63"} Jul 30 19:25:46.627168 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:46.627148 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-896dfc6b7-bck54" Jul 30 19:25:46.628174 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:46.627587 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-77c974ffcf-zvqfw" event={"ID":"68ac40f9-2822-4b96-8fb5-7d37d192bb97","Type":"ContainerStarted","Data":"dfac691d96cbd43bffcd3847e68a44ce3c8ce165e5cee86dc8123c37fe6b3339"} Jul 30 19:25:46.640988 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:46.640928 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vxwhr" podStartSLOduration=67.050902892 podStartE2EDuration="1m8.640910959s" totalImagesPullingTime="1.590008067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 19:24:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:25:46.639511915 +0000 UTC m=+68.859070829" watchObservedRunningTime="2026-07-30 19:25:46.640910959 +0000 UTC m=+68.860469873" Jul 30 19:25:46.682901 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:46.682811 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-896dfc6b7-bck54"] Jul 30 19:25:46.687538 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:46.687507 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-896dfc6b7-bck54"] Jul 30 19:25:46.713818 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:46.713062 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-bbb6865cb-vjk5x" podStartSLOduration=1.713043401 podStartE2EDuration="1.713043401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 19:25:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:25:46.712650469 +0000 UTC m=+68.932209384" watchObservedRunningTime="2026-07-30 19:25:46.713043401 +0000 UTC m=+68.932602319" Jul 30 19:25:46.915177 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:46.915142 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-jz9fk"] Jul 30 19:25:46.926122 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:46.926093 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-jz9fk" Jul 30 19:25:46.929256 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:46.928807 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-jz9fk"] Jul 30 19:25:46.929256 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:46.929095 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-zvzcb\"" Jul 30 19:25:46.929256 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:46.929211 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 30 19:25:46.998150 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:46.998110 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d1d2af65-5934-4636-8c1c-734d05a36186-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-jz9fk\" (UID: \"d1d2af65-5934-4636-8c1c-734d05a36186\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-jz9fk" Jul 30 19:25:47.098987 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:47.098944 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d1d2af65-5934-4636-8c1c-734d05a36186-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-jz9fk\" (UID: \"d1d2af65-5934-4636-8c1c-734d05a36186\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-jz9fk" Jul 30 19:25:47.099544 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:47.099136 2556 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 30 19:25:47.099544 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:25:47.099214 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1d2af65-5934-4636-8c1c-734d05a36186-tls-certificates podName:d1d2af65-5934-4636-8c1c-734d05a36186 nodeName:}" failed. No retries permitted until 2026-07-30 19:25:47.599192316 +0000 UTC m=+69.818751210 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/d1d2af65-5934-4636-8c1c-734d05a36186-tls-certificates") pod "prometheus-operator-admission-webhook-66685ff6d4-jz9fk" (UID: "d1d2af65-5934-4636-8c1c-734d05a36186") : secret "prometheus-operator-admission-webhook-tls" not found Jul 30 19:25:47.603863 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:47.603826 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d1d2af65-5934-4636-8c1c-734d05a36186-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-jz9fk\" (UID: \"d1d2af65-5934-4636-8c1c-734d05a36186\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-jz9fk" Jul 30 19:25:47.615985 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:47.614900 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d1d2af65-5934-4636-8c1c-734d05a36186-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-jz9fk\" (UID: \"d1d2af65-5934-4636-8c1c-734d05a36186\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-jz9fk" Jul 30 19:25:47.839375 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:47.839341 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-jz9fk" Jul 30 19:25:48.324246 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:48.324021 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65ac268e-0120-47af-afc7-4f669f07facd" path="/var/lib/kubelet/pods/65ac268e-0120-47af-afc7-4f669f07facd/volumes" Jul 30 19:25:50.884701 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:50.884664 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-84fc9cc987-hm25f"] Jul 30 19:25:50.891009 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:50.890986 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84fc9cc987-hm25f" Jul 30 19:25:50.893332 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:50.893304 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 30 19:25:50.894679 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:50.894625 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 30 19:25:50.894793 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:50.894676 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 30 19:25:50.894793 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:50.894736 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-smf57\"" Jul 30 19:25:50.894793 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:50.894682 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 30 19:25:50.894962 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:50.894891 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 30 19:25:50.897899 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:50.897874 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84fc9cc987-hm25f"] Jul 30 19:25:51.033588 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:51.033549 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e3d5ab2-a94c-46e8-8347-baa3755abb91-console-serving-cert\") pod \"console-84fc9cc987-hm25f\" (UID: \"1e3d5ab2-a94c-46e8-8347-baa3755abb91\") " pod="openshift-console/console-84fc9cc987-hm25f" Jul 30 19:25:51.033778 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:51.033624 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1e3d5ab2-a94c-46e8-8347-baa3755abb91-service-ca\") pod \"console-84fc9cc987-hm25f\" (UID: \"1e3d5ab2-a94c-46e8-8347-baa3755abb91\") " pod="openshift-console/console-84fc9cc987-hm25f" Jul 30 19:25:51.033778 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:51.033661 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbpmw\" (UniqueName: \"kubernetes.io/projected/1e3d5ab2-a94c-46e8-8347-baa3755abb91-kube-api-access-zbpmw\") pod \"console-84fc9cc987-hm25f\" (UID: \"1e3d5ab2-a94c-46e8-8347-baa3755abb91\") " pod="openshift-console/console-84fc9cc987-hm25f" Jul 30 19:25:51.033778 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:51.033691 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1e3d5ab2-a94c-46e8-8347-baa3755abb91-console-config\") pod \"console-84fc9cc987-hm25f\" (UID: \"1e3d5ab2-a94c-46e8-8347-baa3755abb91\") " pod="openshift-console/console-84fc9cc987-hm25f" Jul 30 19:25:51.033778 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:51.033710 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1e3d5ab2-a94c-46e8-8347-baa3755abb91-oauth-serving-cert\") pod \"console-84fc9cc987-hm25f\" (UID: \"1e3d5ab2-a94c-46e8-8347-baa3755abb91\") " pod="openshift-console/console-84fc9cc987-hm25f" Jul 30 19:25:51.033778 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:51.033744 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1e3d5ab2-a94c-46e8-8347-baa3755abb91-console-oauth-config\") pod \"console-84fc9cc987-hm25f\" (UID: \"1e3d5ab2-a94c-46e8-8347-baa3755abb91\") " pod="openshift-console/console-84fc9cc987-hm25f" Jul 30 19:25:51.135009 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:51.134924 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1e3d5ab2-a94c-46e8-8347-baa3755abb91-console-config\") pod \"console-84fc9cc987-hm25f\" (UID: \"1e3d5ab2-a94c-46e8-8347-baa3755abb91\") " pod="openshift-console/console-84fc9cc987-hm25f" Jul 30 19:25:51.135009 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:51.134966 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1e3d5ab2-a94c-46e8-8347-baa3755abb91-oauth-serving-cert\") pod \"console-84fc9cc987-hm25f\" (UID: \"1e3d5ab2-a94c-46e8-8347-baa3755abb91\") " pod="openshift-console/console-84fc9cc987-hm25f" Jul 30 19:25:51.135217 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:51.135069 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1e3d5ab2-a94c-46e8-8347-baa3755abb91-console-oauth-config\") pod \"console-84fc9cc987-hm25f\" (UID: \"1e3d5ab2-a94c-46e8-8347-baa3755abb91\") " pod="openshift-console/console-84fc9cc987-hm25f" Jul 30 19:25:51.135217 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:51.135134 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e3d5ab2-a94c-46e8-8347-baa3755abb91-console-serving-cert\") pod \"console-84fc9cc987-hm25f\" (UID: \"1e3d5ab2-a94c-46e8-8347-baa3755abb91\") " pod="openshift-console/console-84fc9cc987-hm25f" Jul 30 19:25:51.135217 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:51.135193 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1e3d5ab2-a94c-46e8-8347-baa3755abb91-service-ca\") pod \"console-84fc9cc987-hm25f\" (UID: \"1e3d5ab2-a94c-46e8-8347-baa3755abb91\") " pod="openshift-console/console-84fc9cc987-hm25f" Jul 30 19:25:51.135399 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:51.135231 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbpmw\" (UniqueName: \"kubernetes.io/projected/1e3d5ab2-a94c-46e8-8347-baa3755abb91-kube-api-access-zbpmw\") pod \"console-84fc9cc987-hm25f\" (UID: \"1e3d5ab2-a94c-46e8-8347-baa3755abb91\") " pod="openshift-console/console-84fc9cc987-hm25f" Jul 30 19:25:51.135719 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:51.135694 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1e3d5ab2-a94c-46e8-8347-baa3755abb91-console-config\") pod \"console-84fc9cc987-hm25f\" (UID: \"1e3d5ab2-a94c-46e8-8347-baa3755abb91\") " pod="openshift-console/console-84fc9cc987-hm25f" Jul 30 19:25:51.136240 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:51.136211 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1e3d5ab2-a94c-46e8-8347-baa3755abb91-oauth-serving-cert\") pod \"console-84fc9cc987-hm25f\" (UID: \"1e3d5ab2-a94c-46e8-8347-baa3755abb91\") " pod="openshift-console/console-84fc9cc987-hm25f" Jul 30 19:25:51.136505 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:51.136485 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1e3d5ab2-a94c-46e8-8347-baa3755abb91-service-ca\") pod \"console-84fc9cc987-hm25f\" (UID: \"1e3d5ab2-a94c-46e8-8347-baa3755abb91\") " pod="openshift-console/console-84fc9cc987-hm25f" Jul 30 19:25:51.142063 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:51.140146 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1e3d5ab2-a94c-46e8-8347-baa3755abb91-console-oauth-config\") pod \"console-84fc9cc987-hm25f\" (UID: \"1e3d5ab2-a94c-46e8-8347-baa3755abb91\") " pod="openshift-console/console-84fc9cc987-hm25f" Jul 30 19:25:51.142063 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:51.140822 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e3d5ab2-a94c-46e8-8347-baa3755abb91-console-serving-cert\") pod \"console-84fc9cc987-hm25f\" (UID: \"1e3d5ab2-a94c-46e8-8347-baa3755abb91\") " pod="openshift-console/console-84fc9cc987-hm25f" Jul 30 19:25:51.143675 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:51.143654 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbpmw\" (UniqueName: \"kubernetes.io/projected/1e3d5ab2-a94c-46e8-8347-baa3755abb91-kube-api-access-zbpmw\") pod \"console-84fc9cc987-hm25f\" (UID: \"1e3d5ab2-a94c-46e8-8347-baa3755abb91\") " pod="openshift-console/console-84fc9cc987-hm25f" Jul 30 19:25:51.203585 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:51.203561 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84fc9cc987-hm25f" Jul 30 19:25:53.451675 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:53.451598 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84fc9cc987-hm25f"] Jul 30 19:25:53.457272 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:25:53.457230 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e3d5ab2_a94c_46e8_8347_baa3755abb91.slice/crio-2b9ab9bb49f12909c09ed3bdd1e5c2f91a7c6b9a90c480e06424521d6655365b WatchSource:0}: Error finding container 2b9ab9bb49f12909c09ed3bdd1e5c2f91a7c6b9a90c480e06424521d6655365b: Status 404 returned error can't find the container with id 2b9ab9bb49f12909c09ed3bdd1e5c2f91a7c6b9a90c480e06424521d6655365b Jul 30 19:25:53.469324 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:53.469301 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-jz9fk"] Jul 30 19:25:53.476894 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:25:53.474677 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1d2af65_5934_4636_8c1c_734d05a36186.slice/crio-29070607dca9263f7c8d5eb7c20b5886a1a7012739ebb5a39bd9717255448189 WatchSource:0}: Error finding container 29070607dca9263f7c8d5eb7c20b5886a1a7012739ebb5a39bd9717255448189: Status 404 returned error can't find the container with id 29070607dca9263f7c8d5eb7c20b5886a1a7012739ebb5a39bd9717255448189 Jul 30 19:25:53.660123 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:53.660080 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-97f775b68-lhfdc" event={"ID":"7f32d523-d4ca-4137-a78b-283cc30e0b2b","Type":"ContainerStarted","Data":"30ef70970a20d74cf4b165c2115d28ff9c59222e8d3eb3070588f20ee632567e"} Jul 30 19:25:53.660545 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:53.660330 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-97f775b68-lhfdc" Jul 30 19:25:53.661531 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:53.661500 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84fc9cc987-hm25f" event={"ID":"1e3d5ab2-a94c-46e8-8347-baa3755abb91","Type":"ContainerStarted","Data":"2b9ab9bb49f12909c09ed3bdd1e5c2f91a7c6b9a90c480e06424521d6655365b"} Jul 30 19:25:53.662317 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:53.662291 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-97f775b68-lhfdc" Jul 30 19:25:53.663252 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:53.663220 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-czm5g" event={"ID":"c96c7638-b6ce-4a98-a49d-1bf0fb43edf8","Type":"ContainerStarted","Data":"e629669376c87eff6ef3112679a5c8943406ad3016c63329e022ca510fabe721"} Jul 30 19:25:53.663352 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:53.663321 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-czm5g" Jul 30 19:25:53.664751 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:53.664724 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6tpv8" event={"ID":"8f32c8f1-7bf6-4b84-8205-555365c55861","Type":"ContainerStarted","Data":"6838800643f22fa5de609e955959df26337ce5290da0e3034afe6bf232886ec9"} Jul 30 19:25:53.667170 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:53.666770 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-f2czs" event={"ID":"52b30ff5-e40a-45b5-b6bb-80483bb83057","Type":"ContainerStarted","Data":"2bb6247805c6f85d583ba87e94f7929d0913d1f00e6edd07807b7a1b4e129ec1"} Jul 30 19:25:53.667170 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:53.666827 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-f2czs" event={"ID":"52b30ff5-e40a-45b5-b6bb-80483bb83057","Type":"ContainerStarted","Data":"d9457063f6b9ce7f06d50867404e1461257643874b34fb28c323334f736025ef"} Jul 30 19:25:53.667170 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:53.666857 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-f2czs" Jul 30 19:25:53.668030 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:53.668005 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-jz9fk" event={"ID":"d1d2af65-5934-4636-8c1c-734d05a36186","Type":"ContainerStarted","Data":"29070607dca9263f7c8d5eb7c20b5886a1a7012739ebb5a39bd9717255448189"} Jul 30 19:25:53.676858 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:53.676822 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-97f775b68-lhfdc" podStartSLOduration=2.131800942 podStartE2EDuration="9.676810577s" totalImagesPullingTime="7.545009635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 19:25:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:25:53.675950928 +0000 UTC m=+75.895509842" watchObservedRunningTime="2026-07-30 19:25:53.676810577 +0000 UTC m=+75.896369469" Jul 30 19:25:53.690470 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:53.690427 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-czm5g" podStartSLOduration=66.93472649 podStartE2EDuration="1m15.69041373s" totalImagesPullingTime="8.75568724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 19:24:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:25:53.690146604 +0000 UTC m=+75.909705520" watchObservedRunningTime="2026-07-30 19:25:53.69041373 +0000 UTC m=+75.909972643" Jul 30 19:25:53.705702 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:53.705661 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6tpv8" podStartSLOduration=65.929333025 podStartE2EDuration="1m14.705647979s" totalImagesPullingTime="8.776314954s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 19:24:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:25:53.705159093 +0000 UTC m=+75.924718008" watchObservedRunningTime="2026-07-30 19:25:53.705647979 +0000 UTC m=+75.925206886" Jul 30 19:25:53.721208 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:53.721160 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-f2czs" podStartSLOduration=33.558179767 podStartE2EDuration="40.721151444s" totalImagesPullingTime="7.162971677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 19:25:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:25:53.720457982 +0000 UTC m=+75.940016897" watchObservedRunningTime="2026-07-30 19:25:53.721151444 +0000 UTC m=+75.940710357" Jul 30 19:25:54.675617 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:54.675450 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-jz9fk" event={"ID":"d1d2af65-5934-4636-8c1c-734d05a36186","Type":"ContainerStarted","Data":"017e71276a8868cac06604fa9408496e5054626114856306dc26b14b060c5c42"} Jul 30 19:25:54.676016 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:54.675886 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-jz9fk" Jul 30 19:25:54.676946 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:54.676916 2556 patch_prober.go:28] interesting pod/prometheus-operator-admission-webhook-66685ff6d4-jz9fk container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.133.0.13:8443/healthz\": dial tcp 10.133.0.13:8443: connect: connection refused" start-of-body= Jul 30 19:25:54.677050 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:54.676966 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-jz9fk" podUID="d1d2af65-5934-4636-8c1c-734d05a36186" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.133.0.13:8443/healthz\": dial tcp 10.133.0.13:8443: connect: connection refused" Jul 30 19:25:54.693426 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:54.693253 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-jz9fk" podStartSLOduration=7.567310323 podStartE2EDuration="8.693234153s" totalImagesPullingTime="1.12592383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 19:25:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:25:54.691564643 +0000 UTC m=+76.911123559" watchObservedRunningTime="2026-07-30 19:25:54.693234153 +0000 UTC m=+76.912793047" Jul 30 19:25:55.685208 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:55.685176 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-jz9fk" Jul 30 19:25:55.977887 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:55.977790 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-5wjvs"] Jul 30 19:25:55.980814 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:55.980791 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-758d7cb874-5wjvs" Jul 30 19:25:55.983367 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:55.983246 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 30 19:25:55.984222 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:55.984195 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 30 19:25:55.984222 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:55.984211 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 30 19:25:55.984375 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:55.984307 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 30 19:25:55.984375 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:55.984211 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-trrqn\"" Jul 30 19:25:55.984510 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:55.984440 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 30 19:25:55.991148 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:55.991123 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-5wjvs"] Jul 30 19:25:56.075749 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:56.075699 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5fe53e84-a2ba-4338-8d98-162bc6a708d9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-5wjvs\" (UID: \"5fe53e84-a2ba-4338-8d98-162bc6a708d9\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-5wjvs" Jul 30 19:25:56.075749 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:56.075753 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t4vg\" (UniqueName: \"kubernetes.io/projected/5fe53e84-a2ba-4338-8d98-162bc6a708d9-kube-api-access-5t4vg\") pod \"prometheus-operator-758d7cb874-5wjvs\" (UID: \"5fe53e84-a2ba-4338-8d98-162bc6a708d9\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-5wjvs" Jul 30 19:25:56.075989 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:56.075823 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5fe53e84-a2ba-4338-8d98-162bc6a708d9-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-5wjvs\" (UID: \"5fe53e84-a2ba-4338-8d98-162bc6a708d9\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-5wjvs" Jul 30 19:25:56.075989 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:56.075879 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5fe53e84-a2ba-4338-8d98-162bc6a708d9-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-5wjvs\" (UID: \"5fe53e84-a2ba-4338-8d98-162bc6a708d9\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-5wjvs" Jul 30 19:25:56.177078 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:56.177036 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5fe53e84-a2ba-4338-8d98-162bc6a708d9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-5wjvs\" (UID: \"5fe53e84-a2ba-4338-8d98-162bc6a708d9\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-5wjvs" Jul 30 19:25:56.177268 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:56.177090 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t4vg\" (UniqueName: \"kubernetes.io/projected/5fe53e84-a2ba-4338-8d98-162bc6a708d9-kube-api-access-5t4vg\") pod \"prometheus-operator-758d7cb874-5wjvs\" (UID: \"5fe53e84-a2ba-4338-8d98-162bc6a708d9\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-5wjvs" Jul 30 19:25:56.177268 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:56.177117 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5fe53e84-a2ba-4338-8d98-162bc6a708d9-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-5wjvs\" (UID: \"5fe53e84-a2ba-4338-8d98-162bc6a708d9\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-5wjvs" Jul 30 19:25:56.177268 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:56.177157 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5fe53e84-a2ba-4338-8d98-162bc6a708d9-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-5wjvs\" (UID: \"5fe53e84-a2ba-4338-8d98-162bc6a708d9\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-5wjvs" Jul 30 19:25:56.177866 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:56.177835 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5fe53e84-a2ba-4338-8d98-162bc6a708d9-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-5wjvs\" (UID: \"5fe53e84-a2ba-4338-8d98-162bc6a708d9\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-5wjvs" Jul 30 19:25:56.180032 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:56.180004 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5fe53e84-a2ba-4338-8d98-162bc6a708d9-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-5wjvs\" (UID: \"5fe53e84-a2ba-4338-8d98-162bc6a708d9\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-5wjvs" Jul 30 19:25:56.180516 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:56.180496 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5fe53e84-a2ba-4338-8d98-162bc6a708d9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-5wjvs\" (UID: \"5fe53e84-a2ba-4338-8d98-162bc6a708d9\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-5wjvs" Jul 30 19:25:56.185488 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:56.185453 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t4vg\" (UniqueName: \"kubernetes.io/projected/5fe53e84-a2ba-4338-8d98-162bc6a708d9-kube-api-access-5t4vg\") pod \"prometheus-operator-758d7cb874-5wjvs\" (UID: \"5fe53e84-a2ba-4338-8d98-162bc6a708d9\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-5wjvs" Jul 30 19:25:56.293433 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:56.293335 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-758d7cb874-5wjvs" Jul 30 19:25:56.673377 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:56.673341 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-5wjvs"] Jul 30 19:25:56.677483 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:25:56.677453 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fe53e84_a2ba_4338_8d98_162bc6a708d9.slice/crio-bb9681b7b409f3e8a019d4e1d4fd90ee8b0a1e804860a0ea73d9c7644a2d2682 WatchSource:0}: Error finding container bb9681b7b409f3e8a019d4e1d4fd90ee8b0a1e804860a0ea73d9c7644a2d2682: Status 404 returned error can't find the container with id bb9681b7b409f3e8a019d4e1d4fd90ee8b0a1e804860a0ea73d9c7644a2d2682 Jul 30 19:25:56.686173 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:56.686141 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84fc9cc987-hm25f" event={"ID":"1e3d5ab2-a94c-46e8-8347-baa3755abb91","Type":"ContainerStarted","Data":"60d54341d14fd8703789d8e57196a7f40a835a1d58b22be6ddc640fc9431cb41"} Jul 30 19:25:56.687599 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:56.687574 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-5wjvs" event={"ID":"5fe53e84-a2ba-4338-8d98-162bc6a708d9","Type":"ContainerStarted","Data":"bb9681b7b409f3e8a019d4e1d4fd90ee8b0a1e804860a0ea73d9c7644a2d2682"} Jul 30 19:25:56.703076 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:25:56.703028 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-84fc9cc987-hm25f" podStartSLOduration=3.59733779 podStartE2EDuration="6.703010032s" totalImagesPullingTime="3.105672242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 19:25:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:25:56.701593367 +0000 UTC m=+78.921152280" watchObservedRunningTime="2026-07-30 19:25:56.703010032 +0000 UTC m=+78.922568944" Jul 30 19:26:01.204185 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:01.204038 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-84fc9cc987-hm25f" Jul 30 19:26:01.204185 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:01.204124 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84fc9cc987-hm25f" Jul 30 19:26:01.209041 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:01.209018 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-84fc9cc987-hm25f" Jul 30 19:26:01.709267 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:01.709238 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-84fc9cc987-hm25f" Jul 30 19:26:03.678098 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:03.678064 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-f2czs" Jul 30 19:26:05.723951 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:05.723884 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-77c974ffcf-zvqfw" event={"ID":"68ac40f9-2822-4b96-8fb5-7d37d192bb97","Type":"ContainerStarted","Data":"7d76da5b5f6d0ef47033d6b07bd98ba08af3652250b7f3ebf4362cedbf783bfb"} Jul 30 19:26:05.723951 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:05.723956 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-77c974ffcf-zvqfw" Jul 30 19:26:05.726149 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:05.726123 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-5wjvs" event={"ID":"5fe53e84-a2ba-4338-8d98-162bc6a708d9","Type":"ContainerStarted","Data":"1cee5075a6b62c74f5378b81b2d64b7915fdd3814664e2668b8f5545fa449b59"} Jul 30 19:26:05.726270 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:05.726171 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-5wjvs" event={"ID":"5fe53e84-a2ba-4338-8d98-162bc6a708d9","Type":"ContainerStarted","Data":"858559130b635a54ff5ae6badfa5ebd8bdbc60ad8ab899ab2b5f02e12a4a881b"} Jul 30 19:26:05.738534 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:05.738509 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-77c974ffcf-zvqfw" Jul 30 19:26:05.741740 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:05.741696 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-77c974ffcf-zvqfw" podStartSLOduration=1.311379342 podStartE2EDuration="20.741682515s" totalImagesPullingTime="19.430303173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 19:25:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:26:05.740586731 +0000 UTC m=+87.960145647" watchObservedRunningTime="2026-07-30 19:26:05.741682515 +0000 UTC m=+87.961241427" Jul 30 19:26:05.757481 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:05.757440 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-758d7cb874-5wjvs" podStartSLOduration=2.112614151 podStartE2EDuration="10.757422139s" totalImagesPullingTime="8.644807988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 19:25:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:26:05.756766976 +0000 UTC m=+87.976325890" watchObservedRunningTime="2026-07-30 19:26:05.757422139 +0000 UTC m=+87.976981053" Jul 30 19:26:07.315417 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.315363 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-bk87q"] Jul 30 19:26:07.339287 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.339254 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-bk87q"] Jul 30 19:26:07.339493 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.339303 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ctcb9"] Jul 30 19:26:07.339694 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.339667 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-bk87q" Jul 30 19:26:07.342010 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.341978 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 30 19:26:07.342119 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.342038 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 30 19:26:07.342119 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.342051 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-7b8tj\"" Jul 30 19:26:07.356551 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.356530 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-58k7q"] Jul 30 19:26:07.356655 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.356628 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ctcb9" Jul 30 19:26:07.359202 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.359183 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 30 19:26:07.359348 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.359188 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 30 19:26:07.359452 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.359249 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 30 19:26:07.359634 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.359278 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-twtn8\"" Jul 30 19:26:07.382455 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.382434 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-58k7q"] Jul 30 19:26:07.382772 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.382754 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-58k7q" Jul 30 19:26:07.387359 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.387292 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 30 19:26:07.387359 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.387319 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 30 19:26:07.387359 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.387329 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-fndv2\"" Jul 30 19:26:07.387641 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.387618 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 30 19:26:07.472692 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.472657 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/695d4498-ab00-4727-8f9b-7b18314070bc-node-exporter-tls\") pod \"node-exporter-ctcb9\" (UID: \"695d4498-ab00-4727-8f9b-7b18314070bc\") " pod="openshift-monitoring/node-exporter-ctcb9" Jul 30 19:26:07.472692 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.472696 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/695d4498-ab00-4727-8f9b-7b18314070bc-node-exporter-textfile\") pod \"node-exporter-ctcb9\" (UID: \"695d4498-ab00-4727-8f9b-7b18314070bc\") " pod="openshift-monitoring/node-exporter-ctcb9" Jul 30 19:26:07.472909 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.472721 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/695d4498-ab00-4727-8f9b-7b18314070bc-sys\") pod \"node-exporter-ctcb9\" (UID: \"695d4498-ab00-4727-8f9b-7b18314070bc\") " pod="openshift-monitoring/node-exporter-ctcb9" Jul 30 19:26:07.472909 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.472767 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/695d4498-ab00-4727-8f9b-7b18314070bc-node-exporter-wtmp\") pod \"node-exporter-ctcb9\" (UID: \"695d4498-ab00-4727-8f9b-7b18314070bc\") " pod="openshift-monitoring/node-exporter-ctcb9" Jul 30 19:26:07.472909 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.472797 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/695d4498-ab00-4727-8f9b-7b18314070bc-metrics-client-ca\") pod \"node-exporter-ctcb9\" (UID: \"695d4498-ab00-4727-8f9b-7b18314070bc\") " pod="openshift-monitoring/node-exporter-ctcb9" Jul 30 19:26:07.472909 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.472837 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c8414552-763b-40c0-9a01-f4ad8e5dd521-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-bk87q\" (UID: \"c8414552-763b-40c0-9a01-f4ad8e5dd521\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-bk87q" Jul 30 19:26:07.472909 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.472867 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/695d4498-ab00-4727-8f9b-7b18314070bc-node-exporter-accelerators-collector-config\") pod \"node-exporter-ctcb9\" (UID: \"695d4498-ab00-4727-8f9b-7b18314070bc\") " pod="openshift-monitoring/node-exporter-ctcb9" Jul 30 19:26:07.472909 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.472888 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl97k\" (UniqueName: \"kubernetes.io/projected/695d4498-ab00-4727-8f9b-7b18314070bc-kube-api-access-rl97k\") pod \"node-exporter-ctcb9\" (UID: \"695d4498-ab00-4727-8f9b-7b18314070bc\") " pod="openshift-monitoring/node-exporter-ctcb9" Jul 30 19:26:07.473204 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.472971 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p74b9\" (UniqueName: \"kubernetes.io/projected/c8414552-763b-40c0-9a01-f4ad8e5dd521-kube-api-access-p74b9\") pod \"openshift-state-metrics-6bb4d7b4dd-bk87q\" (UID: \"c8414552-763b-40c0-9a01-f4ad8e5dd521\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-bk87q" Jul 30 19:26:07.473204 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.473013 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/695d4498-ab00-4727-8f9b-7b18314070bc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ctcb9\" (UID: \"695d4498-ab00-4727-8f9b-7b18314070bc\") " pod="openshift-monitoring/node-exporter-ctcb9" Jul 30 19:26:07.473204 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.473051 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c8414552-763b-40c0-9a01-f4ad8e5dd521-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-bk87q\" (UID: \"c8414552-763b-40c0-9a01-f4ad8e5dd521\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-bk87q" Jul 30 19:26:07.473204 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.473088 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8414552-763b-40c0-9a01-f4ad8e5dd521-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-bk87q\" (UID: \"c8414552-763b-40c0-9a01-f4ad8e5dd521\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-bk87q" Jul 30 19:26:07.473204 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.473131 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/695d4498-ab00-4727-8f9b-7b18314070bc-root\") pod \"node-exporter-ctcb9\" (UID: \"695d4498-ab00-4727-8f9b-7b18314070bc\") " pod="openshift-monitoring/node-exporter-ctcb9" Jul 30 19:26:07.573747 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.573674 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/695d4498-ab00-4727-8f9b-7b18314070bc-root\") pod \"node-exporter-ctcb9\" (UID: \"695d4498-ab00-4727-8f9b-7b18314070bc\") " pod="openshift-monitoring/node-exporter-ctcb9" Jul 30 19:26:07.573747 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.573722 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/695d4498-ab00-4727-8f9b-7b18314070bc-node-exporter-tls\") pod \"node-exporter-ctcb9\" (UID: \"695d4498-ab00-4727-8f9b-7b18314070bc\") " pod="openshift-monitoring/node-exporter-ctcb9" Jul 30 19:26:07.573948 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.573752 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/35a74185-af33-4abd-9973-94dc2c4e8bb9-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-58k7q\" (UID: \"35a74185-af33-4abd-9973-94dc2c4e8bb9\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-58k7q" Jul 30 19:26:07.573948 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.573780 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/695d4498-ab00-4727-8f9b-7b18314070bc-node-exporter-textfile\") pod \"node-exporter-ctcb9\" (UID: \"695d4498-ab00-4727-8f9b-7b18314070bc\") " pod="openshift-monitoring/node-exporter-ctcb9" Jul 30 19:26:07.573948 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.573798 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/695d4498-ab00-4727-8f9b-7b18314070bc-root\") pod \"node-exporter-ctcb9\" (UID: \"695d4498-ab00-4727-8f9b-7b18314070bc\") " pod="openshift-monitoring/node-exporter-ctcb9" Jul 30 19:26:07.573948 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.573805 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/695d4498-ab00-4727-8f9b-7b18314070bc-sys\") pod \"node-exporter-ctcb9\" (UID: \"695d4498-ab00-4727-8f9b-7b18314070bc\") " pod="openshift-monitoring/node-exporter-ctcb9" Jul 30 19:26:07.573948 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.573859 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/695d4498-ab00-4727-8f9b-7b18314070bc-sys\") pod \"node-exporter-ctcb9\" (UID: \"695d4498-ab00-4727-8f9b-7b18314070bc\") " pod="openshift-monitoring/node-exporter-ctcb9" Jul 30 19:26:07.573948 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.573887 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/695d4498-ab00-4727-8f9b-7b18314070bc-node-exporter-wtmp\") pod \"node-exporter-ctcb9\" (UID: \"695d4498-ab00-4727-8f9b-7b18314070bc\") " pod="openshift-monitoring/node-exporter-ctcb9" Jul 30 19:26:07.573948 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.573924 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/695d4498-ab00-4727-8f9b-7b18314070bc-metrics-client-ca\") pod \"node-exporter-ctcb9\" (UID: \"695d4498-ab00-4727-8f9b-7b18314070bc\") " pod="openshift-monitoring/node-exporter-ctcb9" Jul 30 19:26:07.574290 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.573952 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vkhb\" (UniqueName: \"kubernetes.io/projected/35a74185-af33-4abd-9973-94dc2c4e8bb9-kube-api-access-4vkhb\") pod \"kube-state-metrics-6bd9664c4-58k7q\" (UID: \"35a74185-af33-4abd-9973-94dc2c4e8bb9\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-58k7q" Jul 30 19:26:07.574290 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.573986 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/35a74185-af33-4abd-9973-94dc2c4e8bb9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-58k7q\" (UID: \"35a74185-af33-4abd-9973-94dc2c4e8bb9\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-58k7q" Jul 30 19:26:07.574290 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.574025 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c8414552-763b-40c0-9a01-f4ad8e5dd521-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-bk87q\" (UID: \"c8414552-763b-40c0-9a01-f4ad8e5dd521\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-bk87q" Jul 30 19:26:07.574290 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.574059 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/695d4498-ab00-4727-8f9b-7b18314070bc-node-exporter-accelerators-collector-config\") pod \"node-exporter-ctcb9\" (UID: \"695d4498-ab00-4727-8f9b-7b18314070bc\") " pod="openshift-monitoring/node-exporter-ctcb9" Jul 30 19:26:07.574290 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.574083 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl97k\" (UniqueName: \"kubernetes.io/projected/695d4498-ab00-4727-8f9b-7b18314070bc-kube-api-access-rl97k\") pod \"node-exporter-ctcb9\" (UID: \"695d4498-ab00-4727-8f9b-7b18314070bc\") " pod="openshift-monitoring/node-exporter-ctcb9" Jul 30 19:26:07.574290 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.574110 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/695d4498-ab00-4727-8f9b-7b18314070bc-node-exporter-textfile\") pod \"node-exporter-ctcb9\" (UID: \"695d4498-ab00-4727-8f9b-7b18314070bc\") " pod="openshift-monitoring/node-exporter-ctcb9" Jul 30 19:26:07.574290 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.574141 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p74b9\" (UniqueName: \"kubernetes.io/projected/c8414552-763b-40c0-9a01-f4ad8e5dd521-kube-api-access-p74b9\") pod \"openshift-state-metrics-6bb4d7b4dd-bk87q\" (UID: \"c8414552-763b-40c0-9a01-f4ad8e5dd521\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-bk87q" Jul 30 19:26:07.574290 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.574166 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/695d4498-ab00-4727-8f9b-7b18314070bc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ctcb9\" (UID: \"695d4498-ab00-4727-8f9b-7b18314070bc\") " pod="openshift-monitoring/node-exporter-ctcb9" Jul 30 19:26:07.574290 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.574201 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c8414552-763b-40c0-9a01-f4ad8e5dd521-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-bk87q\" (UID: \"c8414552-763b-40c0-9a01-f4ad8e5dd521\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-bk87q" Jul 30 19:26:07.574290 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.574231 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/35a74185-af33-4abd-9973-94dc2c4e8bb9-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-58k7q\" (UID: \"35a74185-af33-4abd-9973-94dc2c4e8bb9\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-58k7q" Jul 30 19:26:07.574290 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.574258 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/35a74185-af33-4abd-9973-94dc2c4e8bb9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-58k7q\" (UID: \"35a74185-af33-4abd-9973-94dc2c4e8bb9\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-58k7q" Jul 30 19:26:07.574290 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.574290 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8414552-763b-40c0-9a01-f4ad8e5dd521-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-bk87q\" (UID: \"c8414552-763b-40c0-9a01-f4ad8e5dd521\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-bk87q" Jul 30 19:26:07.574991 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.574315 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35a74185-af33-4abd-9973-94dc2c4e8bb9-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-58k7q\" (UID: \"35a74185-af33-4abd-9973-94dc2c4e8bb9\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-58k7q" Jul 30 19:26:07.574991 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.574462 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/695d4498-ab00-4727-8f9b-7b18314070bc-node-exporter-wtmp\") pod \"node-exporter-ctcb9\" (UID: \"695d4498-ab00-4727-8f9b-7b18314070bc\") " pod="openshift-monitoring/node-exporter-ctcb9" Jul 30 19:26:07.574991 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.574604 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/695d4498-ab00-4727-8f9b-7b18314070bc-metrics-client-ca\") pod \"node-exporter-ctcb9\" (UID: \"695d4498-ab00-4727-8f9b-7b18314070bc\") " pod="openshift-monitoring/node-exporter-ctcb9" Jul 30 19:26:07.574991 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.574848 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/695d4498-ab00-4727-8f9b-7b18314070bc-node-exporter-accelerators-collector-config\") pod \"node-exporter-ctcb9\" (UID: \"695d4498-ab00-4727-8f9b-7b18314070bc\") " pod="openshift-monitoring/node-exporter-ctcb9" Jul 30 19:26:07.575484 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.575442 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8414552-763b-40c0-9a01-f4ad8e5dd521-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-bk87q\" (UID: \"c8414552-763b-40c0-9a01-f4ad8e5dd521\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-bk87q" Jul 30 19:26:07.577427 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.577407 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/695d4498-ab00-4727-8f9b-7b18314070bc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ctcb9\" (UID: \"695d4498-ab00-4727-8f9b-7b18314070bc\") " pod="openshift-monitoring/node-exporter-ctcb9" Jul 30 19:26:07.577769 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.577745 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c8414552-763b-40c0-9a01-f4ad8e5dd521-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-bk87q\" (UID: \"c8414552-763b-40c0-9a01-f4ad8e5dd521\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-bk87q" Jul 30 19:26:07.577921 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.577894 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c8414552-763b-40c0-9a01-f4ad8e5dd521-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-bk87q\" (UID: \"c8414552-763b-40c0-9a01-f4ad8e5dd521\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-bk87q" Jul 30 19:26:07.578599 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.578577 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/695d4498-ab00-4727-8f9b-7b18314070bc-node-exporter-tls\") pod \"node-exporter-ctcb9\" (UID: \"695d4498-ab00-4727-8f9b-7b18314070bc\") " pod="openshift-monitoring/node-exporter-ctcb9" Jul 30 19:26:07.583558 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.583375 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl97k\" (UniqueName: \"kubernetes.io/projected/695d4498-ab00-4727-8f9b-7b18314070bc-kube-api-access-rl97k\") pod \"node-exporter-ctcb9\" (UID: \"695d4498-ab00-4727-8f9b-7b18314070bc\") " pod="openshift-monitoring/node-exporter-ctcb9" Jul 30 19:26:07.583558 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.583411 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p74b9\" (UniqueName: \"kubernetes.io/projected/c8414552-763b-40c0-9a01-f4ad8e5dd521-kube-api-access-p74b9\") pod \"openshift-state-metrics-6bb4d7b4dd-bk87q\" (UID: \"c8414552-763b-40c0-9a01-f4ad8e5dd521\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-bk87q" Jul 30 19:26:07.651672 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.651641 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-bk87q" Jul 30 19:26:07.667824 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.667801 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ctcb9" Jul 30 19:26:07.675089 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.675065 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/35a74185-af33-4abd-9973-94dc2c4e8bb9-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-58k7q\" (UID: \"35a74185-af33-4abd-9973-94dc2c4e8bb9\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-58k7q" Jul 30 19:26:07.675197 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.675111 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vkhb\" (UniqueName: \"kubernetes.io/projected/35a74185-af33-4abd-9973-94dc2c4e8bb9-kube-api-access-4vkhb\") pod \"kube-state-metrics-6bd9664c4-58k7q\" (UID: \"35a74185-af33-4abd-9973-94dc2c4e8bb9\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-58k7q" Jul 30 19:26:07.675197 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.675138 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/35a74185-af33-4abd-9973-94dc2c4e8bb9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-58k7q\" (UID: \"35a74185-af33-4abd-9973-94dc2c4e8bb9\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-58k7q" Jul 30 19:26:07.675307 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.675202 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/35a74185-af33-4abd-9973-94dc2c4e8bb9-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-58k7q\" (UID: \"35a74185-af33-4abd-9973-94dc2c4e8bb9\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-58k7q" Jul 30 19:26:07.675307 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.675230 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/35a74185-af33-4abd-9973-94dc2c4e8bb9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-58k7q\" (UID: \"35a74185-af33-4abd-9973-94dc2c4e8bb9\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-58k7q" Jul 30 19:26:07.675307 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.675258 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35a74185-af33-4abd-9973-94dc2c4e8bb9-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-58k7q\" (UID: \"35a74185-af33-4abd-9973-94dc2c4e8bb9\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-58k7q" Jul 30 19:26:07.675524 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.675500 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/35a74185-af33-4abd-9973-94dc2c4e8bb9-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-58k7q\" (UID: \"35a74185-af33-4abd-9973-94dc2c4e8bb9\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-58k7q" Jul 30 19:26:07.675957 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.675933 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35a74185-af33-4abd-9973-94dc2c4e8bb9-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-58k7q\" (UID: \"35a74185-af33-4abd-9973-94dc2c4e8bb9\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-58k7q" Jul 30 19:26:07.676116 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.676089 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/35a74185-af33-4abd-9973-94dc2c4e8bb9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-58k7q\" (UID: \"35a74185-af33-4abd-9973-94dc2c4e8bb9\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-58k7q" Jul 30 19:26:07.678027 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.678000 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/35a74185-af33-4abd-9973-94dc2c4e8bb9-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-58k7q\" (UID: \"35a74185-af33-4abd-9973-94dc2c4e8bb9\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-58k7q" Jul 30 19:26:07.678193 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.678164 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/35a74185-af33-4abd-9973-94dc2c4e8bb9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-58k7q\" (UID: \"35a74185-af33-4abd-9973-94dc2c4e8bb9\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-58k7q" Jul 30 19:26:07.682847 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.682823 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vkhb\" (UniqueName: \"kubernetes.io/projected/35a74185-af33-4abd-9973-94dc2c4e8bb9-kube-api-access-4vkhb\") pod \"kube-state-metrics-6bd9664c4-58k7q\" (UID: \"35a74185-af33-4abd-9973-94dc2c4e8bb9\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-58k7q" Jul 30 19:26:07.696690 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.696669 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-58k7q" Jul 30 19:26:07.714759 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:26:07.714718 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod695d4498_ab00_4727_8f9b_7b18314070bc.slice/crio-539afe4a3efcb5f4fc2aaf9986ada25e202b636e867f0b5db90aa56c91625689 WatchSource:0}: Error finding container 539afe4a3efcb5f4fc2aaf9986ada25e202b636e867f0b5db90aa56c91625689: Status 404 returned error can't find the container with id 539afe4a3efcb5f4fc2aaf9986ada25e202b636e867f0b5db90aa56c91625689 Jul 30 19:26:07.735799 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.735728 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ctcb9" event={"ID":"695d4498-ab00-4727-8f9b-7b18314070bc","Type":"ContainerStarted","Data":"539afe4a3efcb5f4fc2aaf9986ada25e202b636e867f0b5db90aa56c91625689"} Jul 30 19:26:07.832175 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.831643 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-bk87q"] Jul 30 19:26:07.885367 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:07.885337 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-58k7q"] Jul 30 19:26:07.887813 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:26:07.887784 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35a74185_af33_4abd_9973_94dc2c4e8bb9.slice/crio-55749d74e9236b8dc5d6d9a02471f14c050c4bc997f4d2ee2f3a4fb556df7874 WatchSource:0}: Error finding container 55749d74e9236b8dc5d6d9a02471f14c050c4bc997f4d2ee2f3a4fb556df7874: Status 404 returned error can't find the container with id 55749d74e9236b8dc5d6d9a02471f14c050c4bc997f4d2ee2f3a4fb556df7874 Jul 30 19:26:08.645860 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:08.645046 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-bbb6865cb-vjk5x" Jul 30 19:26:08.742580 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:08.742528 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-58k7q" event={"ID":"35a74185-af33-4abd-9973-94dc2c4e8bb9","Type":"ContainerStarted","Data":"55749d74e9236b8dc5d6d9a02471f14c050c4bc997f4d2ee2f3a4fb556df7874"} Jul 30 19:26:08.744770 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:08.744746 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-bk87q" event={"ID":"c8414552-763b-40c0-9a01-f4ad8e5dd521","Type":"ContainerStarted","Data":"86225f3543ac52aa5c37c6db7ce94c3cbb1a95fb9d36b7a030182595e979187c"} Jul 30 19:26:08.744894 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:08.744788 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-bk87q" event={"ID":"c8414552-763b-40c0-9a01-f4ad8e5dd521","Type":"ContainerStarted","Data":"a1681fa0dca44e6a2b3c7118d752c714db571f2cba70765687c3cc3abbf0662d"} Jul 30 19:26:08.744894 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:08.744807 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-bk87q" event={"ID":"c8414552-763b-40c0-9a01-f4ad8e5dd521","Type":"ContainerStarted","Data":"97736bfde8a8d31ce440f0a12910e10c24f76d88504b62f7714ea00c588157d1"} Jul 30 19:26:10.420296 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:10.420265 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-fd44f7995-8wknj"] Jul 30 19:26:10.439322 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:10.439296 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-fd44f7995-8wknj"] Jul 30 19:26:10.439584 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:10.439397 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-fd44f7995-8wknj" Jul 30 19:26:10.442134 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:10.442111 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 30 19:26:10.442258 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:10.442181 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 30 19:26:10.442321 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:10.442252 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-9hj2krqmibkc\"" Jul 30 19:26:10.442633 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:10.442419 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 30 19:26:10.442633 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:10.442469 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 30 19:26:10.442633 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:10.442490 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-bfc79\"" Jul 30 19:26:10.442633 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:10.442419 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 30 19:26:10.602296 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:10.602264 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b0261e45-1760-4a3e-925d-fe0d63dc7635-secret-grpc-tls\") pod \"thanos-querier-fd44f7995-8wknj\" (UID: \"b0261e45-1760-4a3e-925d-fe0d63dc7635\") " pod="openshift-monitoring/thanos-querier-fd44f7995-8wknj" Jul 30 19:26:10.602455 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:10.602320 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b0261e45-1760-4a3e-925d-fe0d63dc7635-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-fd44f7995-8wknj\" (UID: \"b0261e45-1760-4a3e-925d-fe0d63dc7635\") " pod="openshift-monitoring/thanos-querier-fd44f7995-8wknj" Jul 30 19:26:10.602455 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:10.602351 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b0261e45-1760-4a3e-925d-fe0d63dc7635-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-fd44f7995-8wknj\" (UID: \"b0261e45-1760-4a3e-925d-fe0d63dc7635\") " pod="openshift-monitoring/thanos-querier-fd44f7995-8wknj" Jul 30 19:26:10.602455 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:10.602393 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0261e45-1760-4a3e-925d-fe0d63dc7635-metrics-client-ca\") pod \"thanos-querier-fd44f7995-8wknj\" (UID: \"b0261e45-1760-4a3e-925d-fe0d63dc7635\") " pod="openshift-monitoring/thanos-querier-fd44f7995-8wknj" Jul 30 19:26:10.602623 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:10.602496 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b0261e45-1760-4a3e-925d-fe0d63dc7635-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-fd44f7995-8wknj\" (UID: \"b0261e45-1760-4a3e-925d-fe0d63dc7635\") " pod="openshift-monitoring/thanos-querier-fd44f7995-8wknj" Jul 30 19:26:10.602623 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:10.602547 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b0261e45-1760-4a3e-925d-fe0d63dc7635-secret-thanos-querier-tls\") pod \"thanos-querier-fd44f7995-8wknj\" (UID: \"b0261e45-1760-4a3e-925d-fe0d63dc7635\") " pod="openshift-monitoring/thanos-querier-fd44f7995-8wknj" Jul 30 19:26:10.602623 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:10.602571 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jthfc\" (UniqueName: \"kubernetes.io/projected/b0261e45-1760-4a3e-925d-fe0d63dc7635-kube-api-access-jthfc\") pod \"thanos-querier-fd44f7995-8wknj\" (UID: \"b0261e45-1760-4a3e-925d-fe0d63dc7635\") " pod="openshift-monitoring/thanos-querier-fd44f7995-8wknj" Jul 30 19:26:10.602752 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:10.602624 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b0261e45-1760-4a3e-925d-fe0d63dc7635-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-fd44f7995-8wknj\" (UID: \"b0261e45-1760-4a3e-925d-fe0d63dc7635\") " pod="openshift-monitoring/thanos-querier-fd44f7995-8wknj" Jul 30 19:26:10.703270 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:10.703169 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b0261e45-1760-4a3e-925d-fe0d63dc7635-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-fd44f7995-8wknj\" (UID: \"b0261e45-1760-4a3e-925d-fe0d63dc7635\") " pod="openshift-monitoring/thanos-querier-fd44f7995-8wknj" Jul 30 19:26:10.703270 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:10.703218 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b0261e45-1760-4a3e-925d-fe0d63dc7635-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-fd44f7995-8wknj\" (UID: \"b0261e45-1760-4a3e-925d-fe0d63dc7635\") " pod="openshift-monitoring/thanos-querier-fd44f7995-8wknj" Jul 30 19:26:10.703270 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:10.703247 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0261e45-1760-4a3e-925d-fe0d63dc7635-metrics-client-ca\") pod \"thanos-querier-fd44f7995-8wknj\" (UID: \"b0261e45-1760-4a3e-925d-fe0d63dc7635\") " pod="openshift-monitoring/thanos-querier-fd44f7995-8wknj" Jul 30 19:26:10.703583 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:10.703291 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b0261e45-1760-4a3e-925d-fe0d63dc7635-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-fd44f7995-8wknj\" (UID: \"b0261e45-1760-4a3e-925d-fe0d63dc7635\") " pod="openshift-monitoring/thanos-querier-fd44f7995-8wknj" Jul 30 19:26:10.703583 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:10.703335 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b0261e45-1760-4a3e-925d-fe0d63dc7635-secret-thanos-querier-tls\") pod \"thanos-querier-fd44f7995-8wknj\" (UID: \"b0261e45-1760-4a3e-925d-fe0d63dc7635\") " pod="openshift-monitoring/thanos-querier-fd44f7995-8wknj" Jul 30 19:26:10.703583 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:10.703352 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jthfc\" (UniqueName: \"kubernetes.io/projected/b0261e45-1760-4a3e-925d-fe0d63dc7635-kube-api-access-jthfc\") pod \"thanos-querier-fd44f7995-8wknj\" (UID: \"b0261e45-1760-4a3e-925d-fe0d63dc7635\") " pod="openshift-monitoring/thanos-querier-fd44f7995-8wknj" Jul 30 19:26:10.703583 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:10.703486 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b0261e45-1760-4a3e-925d-fe0d63dc7635-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-fd44f7995-8wknj\" (UID: \"b0261e45-1760-4a3e-925d-fe0d63dc7635\") " pod="openshift-monitoring/thanos-querier-fd44f7995-8wknj" Jul 30 19:26:10.703583 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:10.703557 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b0261e45-1760-4a3e-925d-fe0d63dc7635-secret-grpc-tls\") pod \"thanos-querier-fd44f7995-8wknj\" (UID: \"b0261e45-1760-4a3e-925d-fe0d63dc7635\") " pod="openshift-monitoring/thanos-querier-fd44f7995-8wknj" Jul 30 19:26:10.704131 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:10.704101 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0261e45-1760-4a3e-925d-fe0d63dc7635-metrics-client-ca\") pod \"thanos-querier-fd44f7995-8wknj\" (UID: \"b0261e45-1760-4a3e-925d-fe0d63dc7635\") " pod="openshift-monitoring/thanos-querier-fd44f7995-8wknj" Jul 30 19:26:10.706432 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:10.706367 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b0261e45-1760-4a3e-925d-fe0d63dc7635-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-fd44f7995-8wknj\" (UID: \"b0261e45-1760-4a3e-925d-fe0d63dc7635\") " pod="openshift-monitoring/thanos-querier-fd44f7995-8wknj" Jul 30 19:26:10.706585 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:10.706532 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b0261e45-1760-4a3e-925d-fe0d63dc7635-secret-thanos-querier-tls\") pod \"thanos-querier-fd44f7995-8wknj\" (UID: \"b0261e45-1760-4a3e-925d-fe0d63dc7635\") " pod="openshift-monitoring/thanos-querier-fd44f7995-8wknj" Jul 30 19:26:10.706870 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:10.706846 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b0261e45-1760-4a3e-925d-fe0d63dc7635-secret-grpc-tls\") pod \"thanos-querier-fd44f7995-8wknj\" (UID: \"b0261e45-1760-4a3e-925d-fe0d63dc7635\") " pod="openshift-monitoring/thanos-querier-fd44f7995-8wknj" Jul 30 19:26:10.707614 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:10.707589 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b0261e45-1760-4a3e-925d-fe0d63dc7635-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-fd44f7995-8wknj\" (UID: \"b0261e45-1760-4a3e-925d-fe0d63dc7635\") " pod="openshift-monitoring/thanos-querier-fd44f7995-8wknj" Jul 30 19:26:10.707708 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:10.707674 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b0261e45-1760-4a3e-925d-fe0d63dc7635-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-fd44f7995-8wknj\" (UID: \"b0261e45-1760-4a3e-925d-fe0d63dc7635\") " pod="openshift-monitoring/thanos-querier-fd44f7995-8wknj" Jul 30 19:26:10.707769 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:10.707752 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b0261e45-1760-4a3e-925d-fe0d63dc7635-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-fd44f7995-8wknj\" (UID: \"b0261e45-1760-4a3e-925d-fe0d63dc7635\") " pod="openshift-monitoring/thanos-querier-fd44f7995-8wknj" Jul 30 19:26:10.711614 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:10.711593 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jthfc\" (UniqueName: \"kubernetes.io/projected/b0261e45-1760-4a3e-925d-fe0d63dc7635-kube-api-access-jthfc\") pod \"thanos-querier-fd44f7995-8wknj\" (UID: \"b0261e45-1760-4a3e-925d-fe0d63dc7635\") " pod="openshift-monitoring/thanos-querier-fd44f7995-8wknj" Jul 30 19:26:10.751759 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:10.751737 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-fd44f7995-8wknj" Jul 30 19:26:10.969641 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:10.969541 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-fd44f7995-8wknj"] Jul 30 19:26:10.980601 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:26:10.980572 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0261e45_1760_4a3e_925d_fe0d63dc7635.slice/crio-424e52806927df3aefa9d3075bd239e67b1ff27f5c0070f0774b4fea793db546 WatchSource:0}: Error finding container 424e52806927df3aefa9d3075bd239e67b1ff27f5c0070f0774b4fea793db546: Status 404 returned error can't find the container with id 424e52806927df3aefa9d3075bd239e67b1ff27f5c0070f0774b4fea793db546 Jul 30 19:26:11.759256 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:11.759219 2556 generic.go:356] "Generic (PLEG): container finished" podID="695d4498-ab00-4727-8f9b-7b18314070bc" containerID="046a60d29c385ff671bb7db233b68508619cd48ae00afa23992f680084cc28df" exitCode=0 Jul 30 19:26:11.759728 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:11.759323 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ctcb9" event={"ID":"695d4498-ab00-4727-8f9b-7b18314070bc","Type":"ContainerDied","Data":"046a60d29c385ff671bb7db233b68508619cd48ae00afa23992f680084cc28df"} Jul 30 19:26:11.762043 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:11.762013 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-58k7q" event={"ID":"35a74185-af33-4abd-9973-94dc2c4e8bb9","Type":"ContainerStarted","Data":"8c637abd209af5be110fe3eb62f8c7fabea3a2423ef0832b2b966b06aae17226"} Jul 30 19:26:11.762182 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:11.762074 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-58k7q" event={"ID":"35a74185-af33-4abd-9973-94dc2c4e8bb9","Type":"ContainerStarted","Data":"375986dededd3af67317bc5abd3964f8b0540db4e44bf060b7d8873dbbf532c7"} Jul 30 19:26:11.762182 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:11.762105 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-58k7q" event={"ID":"35a74185-af33-4abd-9973-94dc2c4e8bb9","Type":"ContainerStarted","Data":"16ded53675397379b140d45b99573d0c3025f93ddb9af5157498f59869440c81"} Jul 30 19:26:11.764433 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:11.764410 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-bk87q" event={"ID":"c8414552-763b-40c0-9a01-f4ad8e5dd521","Type":"ContainerStarted","Data":"c73f3ff3b4ff81333b006b601979f969131ad7cc9ba1389914c61621922c4c57"} Jul 30 19:26:11.765773 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:11.765749 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-fd44f7995-8wknj" event={"ID":"b0261e45-1760-4a3e-925d-fe0d63dc7635","Type":"ContainerStarted","Data":"424e52806927df3aefa9d3075bd239e67b1ff27f5c0070f0774b4fea793db546"} Jul 30 19:26:11.835215 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:11.834525 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-bk87q" podStartSLOduration=2.155327104 podStartE2EDuration="4.83450486s" totalImagesPullingTime="2.679177756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 19:26:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:26:11.80614999 +0000 UTC m=+94.025708903" watchObservedRunningTime="2026-07-30 19:26:11.83450486 +0000 UTC m=+94.054063775" Jul 30 19:26:11.835215 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:11.834745 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-58k7q" podStartSLOduration=1.919606587 podStartE2EDuration="4.834737062s" totalImagesPullingTime="2.915130475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 19:26:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:26:11.833794795 +0000 UTC m=+94.053353711" watchObservedRunningTime="2026-07-30 19:26:11.834737062 +0000 UTC m=+94.054295976" Jul 30 19:26:12.118845 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:12.118793 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84fc9cc987-hm25f"] Jul 30 19:26:12.120553 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:12.120473 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-hpn6r"] Jul 30 19:26:12.137155 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:12.136554 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-hpn6r"] Jul 30 19:26:12.137155 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:12.136654 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-hpn6r" Jul 30 19:26:12.140982 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:12.140957 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 30 19:26:12.141490 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:12.141293 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-2swpp\"" Jul 30 19:26:12.217095 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:12.217055 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ed0bae37-0a50-4f34-b0b8-bf912e2f75ed-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-hpn6r\" (UID: \"ed0bae37-0a50-4f34-b0b8-bf912e2f75ed\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-hpn6r" Jul 30 19:26:12.217251 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:12.217145 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8k87\" (UniqueName: \"kubernetes.io/projected/ed0bae37-0a50-4f34-b0b8-bf912e2f75ed-kube-api-access-c8k87\") pod \"monitoring-plugin-6dffbbd67-hpn6r\" (UID: \"ed0bae37-0a50-4f34-b0b8-bf912e2f75ed\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-hpn6r" Jul 30 19:26:12.318328 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:12.318232 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8k87\" (UniqueName: \"kubernetes.io/projected/ed0bae37-0a50-4f34-b0b8-bf912e2f75ed-kube-api-access-c8k87\") pod \"monitoring-plugin-6dffbbd67-hpn6r\" (UID: \"ed0bae37-0a50-4f34-b0b8-bf912e2f75ed\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-hpn6r" Jul 30 19:26:12.318328 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:12.318314 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ed0bae37-0a50-4f34-b0b8-bf912e2f75ed-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-hpn6r\" (UID: \"ed0bae37-0a50-4f34-b0b8-bf912e2f75ed\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-hpn6r" Jul 30 19:26:12.321847 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:12.321804 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ed0bae37-0a50-4f34-b0b8-bf912e2f75ed-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-hpn6r\" (UID: \"ed0bae37-0a50-4f34-b0b8-bf912e2f75ed\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-hpn6r" Jul 30 19:26:12.327484 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:12.327461 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8k87\" (UniqueName: \"kubernetes.io/projected/ed0bae37-0a50-4f34-b0b8-bf912e2f75ed-kube-api-access-c8k87\") pod \"monitoring-plugin-6dffbbd67-hpn6r\" (UID: \"ed0bae37-0a50-4f34-b0b8-bf912e2f75ed\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-hpn6r" Jul 30 19:26:12.451821 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:12.451795 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-hpn6r" Jul 30 19:26:12.554852 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:12.554808 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6cbd87958-csgb5"] Jul 30 19:26:12.585085 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:12.583713 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6cbd87958-csgb5"] Jul 30 19:26:12.585085 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:12.583852 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6cbd87958-csgb5" Jul 30 19:26:12.588052 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:12.587314 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 30 19:26:12.588052 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:12.587409 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 30 19:26:12.588052 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:12.587557 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 30 19:26:12.588052 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:12.587849 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 30 19:26:12.588052 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:12.587893 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 30 19:26:12.588052 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:12.587851 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-sf487\"" Jul 30 19:26:12.598399 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:12.598352 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 30 19:26:12.620509 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:12.620486 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c71030a2-fcdd-466c-8299-01c7f963c64d-secret-telemeter-client\") pod \"telemeter-client-6cbd87958-csgb5\" (UID: \"c71030a2-fcdd-466c-8299-01c7f963c64d\") " pod="openshift-monitoring/telemeter-client-6cbd87958-csgb5" Jul 30 19:26:12.620638 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:12.620611 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvhk7\" (UniqueName: \"kubernetes.io/projected/c71030a2-fcdd-466c-8299-01c7f963c64d-kube-api-access-wvhk7\") pod \"telemeter-client-6cbd87958-csgb5\" (UID: \"c71030a2-fcdd-466c-8299-01c7f963c64d\") " pod="openshift-monitoring/telemeter-client-6cbd87958-csgb5" Jul 30 19:26:12.620758 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:12.620679 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c71030a2-fcdd-466c-8299-01c7f963c64d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6cbd87958-csgb5\" (UID: \"c71030a2-fcdd-466c-8299-01c7f963c64d\") " pod="openshift-monitoring/telemeter-client-6cbd87958-csgb5" Jul 30 19:26:12.620758 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:12.620725 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c71030a2-fcdd-466c-8299-01c7f963c64d-serving-certs-ca-bundle\") pod \"telemeter-client-6cbd87958-csgb5\" (UID: \"c71030a2-fcdd-466c-8299-01c7f963c64d\") " pod="openshift-monitoring/telemeter-client-6cbd87958-csgb5" Jul 30 19:26:12.620865 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:12.620809 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c71030a2-fcdd-466c-8299-01c7f963c64d-federate-client-tls\") pod \"telemeter-client-6cbd87958-csgb5\" (UID: \"c71030a2-fcdd-466c-8299-01c7f963c64d\") " pod="openshift-monitoring/telemeter-client-6cbd87958-csgb5" Jul 30 19:26:12.620865 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:12.620852 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c71030a2-fcdd-466c-8299-01c7f963c64d-metrics-client-ca\") pod \"telemeter-client-6cbd87958-csgb5\" (UID: \"c71030a2-fcdd-466c-8299-01c7f963c64d\") " pod="openshift-monitoring/telemeter-client-6cbd87958-csgb5" Jul 30 19:26:12.620955 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:12.620917 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c71030a2-fcdd-466c-8299-01c7f963c64d-telemeter-client-tls\") pod \"telemeter-client-6cbd87958-csgb5\" (UID: \"c71030a2-fcdd-466c-8299-01c7f963c64d\") " pod="openshift-monitoring/telemeter-client-6cbd87958-csgb5" Jul 30 19:26:12.620955 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:12.620944 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c71030a2-fcdd-466c-8299-01c7f963c64d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6cbd87958-csgb5\" (UID: \"c71030a2-fcdd-466c-8299-01c7f963c64d\") " pod="openshift-monitoring/telemeter-client-6cbd87958-csgb5" Jul 30 19:26:12.647471 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:12.647411 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-hpn6r"] Jul 30 19:26:12.651473 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:26:12.651445 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded0bae37_0a50_4f34_b0b8_bf912e2f75ed.slice/crio-942feac971e2b3478305fe17daa401697b0767ab52cad33987e9658c3726fdcf WatchSource:0}: Error finding container 942feac971e2b3478305fe17daa401697b0767ab52cad33987e9658c3726fdcf: Status 404 returned error can't find the container with id 942feac971e2b3478305fe17daa401697b0767ab52cad33987e9658c3726fdcf Jul 30 19:26:12.721784 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:12.721755 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c71030a2-fcdd-466c-8299-01c7f963c64d-metrics-client-ca\") pod \"telemeter-client-6cbd87958-csgb5\" (UID: \"c71030a2-fcdd-466c-8299-01c7f963c64d\") " pod="openshift-monitoring/telemeter-client-6cbd87958-csgb5" Jul 30 19:26:12.721906 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:12.721815 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c71030a2-fcdd-466c-8299-01c7f963c64d-telemeter-client-tls\") pod \"telemeter-client-6cbd87958-csgb5\" (UID: \"c71030a2-fcdd-466c-8299-01c7f963c64d\") " pod="openshift-monitoring/telemeter-client-6cbd87958-csgb5" Jul 30 19:26:12.721906 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:12.721835 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c71030a2-fcdd-466c-8299-01c7f963c64d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6cbd87958-csgb5\" (UID: \"c71030a2-fcdd-466c-8299-01c7f963c64d\") " pod="openshift-monitoring/telemeter-client-6cbd87958-csgb5" Jul 30 19:26:12.721906 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:12.721862 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c71030a2-fcdd-466c-8299-01c7f963c64d-secret-telemeter-client\") pod \"telemeter-client-6cbd87958-csgb5\" (UID: \"c71030a2-fcdd-466c-8299-01c7f963c64d\") " pod="openshift-monitoring/telemeter-client-6cbd87958-csgb5" Jul 30 19:26:12.721906 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:12.721886 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvhk7\" (UniqueName: \"kubernetes.io/projected/c71030a2-fcdd-466c-8299-01c7f963c64d-kube-api-access-wvhk7\") pod \"telemeter-client-6cbd87958-csgb5\" (UID: \"c71030a2-fcdd-466c-8299-01c7f963c64d\") " pod="openshift-monitoring/telemeter-client-6cbd87958-csgb5" Jul 30 19:26:12.722093 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:12.721932 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c71030a2-fcdd-466c-8299-01c7f963c64d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6cbd87958-csgb5\" (UID: \"c71030a2-fcdd-466c-8299-01c7f963c64d\") " pod="openshift-monitoring/telemeter-client-6cbd87958-csgb5" Jul 30 19:26:12.722093 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:12.721975 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c71030a2-fcdd-466c-8299-01c7f963c64d-serving-certs-ca-bundle\") pod \"telemeter-client-6cbd87958-csgb5\" (UID: \"c71030a2-fcdd-466c-8299-01c7f963c64d\") " pod="openshift-monitoring/telemeter-client-6cbd87958-csgb5" Jul 30 19:26:12.722093 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:12.722011 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c71030a2-fcdd-466c-8299-01c7f963c64d-federate-client-tls\") pod \"telemeter-client-6cbd87958-csgb5\" (UID: \"c71030a2-fcdd-466c-8299-01c7f963c64d\") " pod="openshift-monitoring/telemeter-client-6cbd87958-csgb5" Jul 30 19:26:12.723139 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:12.722569 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c71030a2-fcdd-466c-8299-01c7f963c64d-metrics-client-ca\") pod \"telemeter-client-6cbd87958-csgb5\" (UID: \"c71030a2-fcdd-466c-8299-01c7f963c64d\") " pod="openshift-monitoring/telemeter-client-6cbd87958-csgb5" Jul 30 19:26:12.723139 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:12.722872 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c71030a2-fcdd-466c-8299-01c7f963c64d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6cbd87958-csgb5\" (UID: \"c71030a2-fcdd-466c-8299-01c7f963c64d\") " pod="openshift-monitoring/telemeter-client-6cbd87958-csgb5" Jul 30 19:26:12.723626 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:12.723564 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c71030a2-fcdd-466c-8299-01c7f963c64d-serving-certs-ca-bundle\") pod \"telemeter-client-6cbd87958-csgb5\" (UID: \"c71030a2-fcdd-466c-8299-01c7f963c64d\") " pod="openshift-monitoring/telemeter-client-6cbd87958-csgb5" Jul 30 19:26:12.725067 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:12.725041 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c71030a2-fcdd-466c-8299-01c7f963c64d-federate-client-tls\") pod \"telemeter-client-6cbd87958-csgb5\" (UID: \"c71030a2-fcdd-466c-8299-01c7f963c64d\") " pod="openshift-monitoring/telemeter-client-6cbd87958-csgb5" Jul 30 19:26:12.725237 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:12.725201 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c71030a2-fcdd-466c-8299-01c7f963c64d-telemeter-client-tls\") pod \"telemeter-client-6cbd87958-csgb5\" (UID: \"c71030a2-fcdd-466c-8299-01c7f963c64d\") " pod="openshift-monitoring/telemeter-client-6cbd87958-csgb5" Jul 30 19:26:12.725972 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:12.725948 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c71030a2-fcdd-466c-8299-01c7f963c64d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6cbd87958-csgb5\" (UID: \"c71030a2-fcdd-466c-8299-01c7f963c64d\") " pod="openshift-monitoring/telemeter-client-6cbd87958-csgb5" Jul 30 19:26:12.726056 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:12.725955 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c71030a2-fcdd-466c-8299-01c7f963c64d-secret-telemeter-client\") pod \"telemeter-client-6cbd87958-csgb5\" (UID: \"c71030a2-fcdd-466c-8299-01c7f963c64d\") " pod="openshift-monitoring/telemeter-client-6cbd87958-csgb5" Jul 30 19:26:12.730625 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:12.730600 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvhk7\" (UniqueName: \"kubernetes.io/projected/c71030a2-fcdd-466c-8299-01c7f963c64d-kube-api-access-wvhk7\") pod \"telemeter-client-6cbd87958-csgb5\" (UID: \"c71030a2-fcdd-466c-8299-01c7f963c64d\") " pod="openshift-monitoring/telemeter-client-6cbd87958-csgb5" Jul 30 19:26:12.774464 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:12.774431 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-hpn6r" event={"ID":"ed0bae37-0a50-4f34-b0b8-bf912e2f75ed","Type":"ContainerStarted","Data":"942feac971e2b3478305fe17daa401697b0767ab52cad33987e9658c3726fdcf"} Jul 30 19:26:12.776968 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:12.776901 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ctcb9" event={"ID":"695d4498-ab00-4727-8f9b-7b18314070bc","Type":"ContainerStarted","Data":"a2fad5025b6b1127c566ccdca14b6af56d23295b08bcffa2325e1ed61c0bc4e4"} Jul 30 19:26:12.776968 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:12.776956 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ctcb9" event={"ID":"695d4498-ab00-4727-8f9b-7b18314070bc","Type":"ContainerStarted","Data":"24cf0920e94982841fd77dce971da3f5d8eed16bede07f8d2ba426c7e188f988"} Jul 30 19:26:12.796339 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:12.796299 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ctcb9" podStartSLOduration=2.559206477 podStartE2EDuration="5.796286264s" totalImagesPullingTime="3.088743201s" totalInitContainerRuntime="148.336586ms" isStatefulPod=true podCreationTimestamp="2026-07-30 19:26:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:26:12.794829649 +0000 UTC m=+95.014388566" watchObservedRunningTime="2026-07-30 19:26:12.796286264 +0000 UTC m=+95.015845177" Jul 30 19:26:12.906924 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:12.906890 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6cbd87958-csgb5" Jul 30 19:26:13.085527 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.085437 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6878d7dcb-gd2n9"] Jul 30 19:26:13.102587 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.102551 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6878d7dcb-gd2n9"] Jul 30 19:26:13.102746 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.102675 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6878d7dcb-gd2n9" Jul 30 19:26:13.111225 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.111201 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 30 19:26:13.126751 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.126723 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/85ee7f74-b251-4c68-b3af-0494b35df222-console-serving-cert\") pod \"console-6878d7dcb-gd2n9\" (UID: \"85ee7f74-b251-4c68-b3af-0494b35df222\") " pod="openshift-console/console-6878d7dcb-gd2n9" Jul 30 19:26:13.126997 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.126771 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85ee7f74-b251-4c68-b3af-0494b35df222-trusted-ca-bundle\") pod \"console-6878d7dcb-gd2n9\" (UID: \"85ee7f74-b251-4c68-b3af-0494b35df222\") " pod="openshift-console/console-6878d7dcb-gd2n9" Jul 30 19:26:13.126997 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.126831 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/85ee7f74-b251-4c68-b3af-0494b35df222-console-config\") pod \"console-6878d7dcb-gd2n9\" (UID: \"85ee7f74-b251-4c68-b3af-0494b35df222\") " pod="openshift-console/console-6878d7dcb-gd2n9" Jul 30 19:26:13.126997 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.126853 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/85ee7f74-b251-4c68-b3af-0494b35df222-service-ca\") pod \"console-6878d7dcb-gd2n9\" (UID: \"85ee7f74-b251-4c68-b3af-0494b35df222\") " pod="openshift-console/console-6878d7dcb-gd2n9" Jul 30 19:26:13.127186 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.127014 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/85ee7f74-b251-4c68-b3af-0494b35df222-console-oauth-config\") pod \"console-6878d7dcb-gd2n9\" (UID: \"85ee7f74-b251-4c68-b3af-0494b35df222\") " pod="openshift-console/console-6878d7dcb-gd2n9" Jul 30 19:26:13.127186 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.127048 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/85ee7f74-b251-4c68-b3af-0494b35df222-oauth-serving-cert\") pod \"console-6878d7dcb-gd2n9\" (UID: \"85ee7f74-b251-4c68-b3af-0494b35df222\") " pod="openshift-console/console-6878d7dcb-gd2n9" Jul 30 19:26:13.127186 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.127075 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjn9z\" (UniqueName: \"kubernetes.io/projected/85ee7f74-b251-4c68-b3af-0494b35df222-kube-api-access-sjn9z\") pod \"console-6878d7dcb-gd2n9\" (UID: \"85ee7f74-b251-4c68-b3af-0494b35df222\") " pod="openshift-console/console-6878d7dcb-gd2n9" Jul 30 19:26:13.233516 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.233404 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/85ee7f74-b251-4c68-b3af-0494b35df222-console-oauth-config\") pod \"console-6878d7dcb-gd2n9\" (UID: \"85ee7f74-b251-4c68-b3af-0494b35df222\") " pod="openshift-console/console-6878d7dcb-gd2n9" Jul 30 19:26:13.233516 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.233452 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/85ee7f74-b251-4c68-b3af-0494b35df222-oauth-serving-cert\") pod \"console-6878d7dcb-gd2n9\" (UID: \"85ee7f74-b251-4c68-b3af-0494b35df222\") " pod="openshift-console/console-6878d7dcb-gd2n9" Jul 30 19:26:13.233516 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.233480 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjn9z\" (UniqueName: \"kubernetes.io/projected/85ee7f74-b251-4c68-b3af-0494b35df222-kube-api-access-sjn9z\") pod \"console-6878d7dcb-gd2n9\" (UID: \"85ee7f74-b251-4c68-b3af-0494b35df222\") " pod="openshift-console/console-6878d7dcb-gd2n9" Jul 30 19:26:13.233967 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.233537 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/85ee7f74-b251-4c68-b3af-0494b35df222-console-serving-cert\") pod \"console-6878d7dcb-gd2n9\" (UID: \"85ee7f74-b251-4c68-b3af-0494b35df222\") " pod="openshift-console/console-6878d7dcb-gd2n9" Jul 30 19:26:13.233967 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.233686 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85ee7f74-b251-4c68-b3af-0494b35df222-trusted-ca-bundle\") pod \"console-6878d7dcb-gd2n9\" (UID: \"85ee7f74-b251-4c68-b3af-0494b35df222\") " pod="openshift-console/console-6878d7dcb-gd2n9" Jul 30 19:26:13.234171 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.233916 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/85ee7f74-b251-4c68-b3af-0494b35df222-console-config\") pod \"console-6878d7dcb-gd2n9\" (UID: \"85ee7f74-b251-4c68-b3af-0494b35df222\") " pod="openshift-console/console-6878d7dcb-gd2n9" Jul 30 19:26:13.234404 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.234362 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/85ee7f74-b251-4c68-b3af-0494b35df222-service-ca\") pod \"console-6878d7dcb-gd2n9\" (UID: \"85ee7f74-b251-4c68-b3af-0494b35df222\") " pod="openshift-console/console-6878d7dcb-gd2n9" Jul 30 19:26:13.234817 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.234785 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/85ee7f74-b251-4c68-b3af-0494b35df222-console-config\") pod \"console-6878d7dcb-gd2n9\" (UID: \"85ee7f74-b251-4c68-b3af-0494b35df222\") " pod="openshift-console/console-6878d7dcb-gd2n9" Jul 30 19:26:13.235602 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.235563 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/85ee7f74-b251-4c68-b3af-0494b35df222-oauth-serving-cert\") pod \"console-6878d7dcb-gd2n9\" (UID: \"85ee7f74-b251-4c68-b3af-0494b35df222\") " pod="openshift-console/console-6878d7dcb-gd2n9" Jul 30 19:26:13.235811 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.235789 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85ee7f74-b251-4c68-b3af-0494b35df222-trusted-ca-bundle\") pod \"console-6878d7dcb-gd2n9\" (UID: \"85ee7f74-b251-4c68-b3af-0494b35df222\") " pod="openshift-console/console-6878d7dcb-gd2n9" Jul 30 19:26:13.238130 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.238109 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/85ee7f74-b251-4c68-b3af-0494b35df222-service-ca\") pod \"console-6878d7dcb-gd2n9\" (UID: \"85ee7f74-b251-4c68-b3af-0494b35df222\") " pod="openshift-console/console-6878d7dcb-gd2n9" Jul 30 19:26:13.238841 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.238771 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/85ee7f74-b251-4c68-b3af-0494b35df222-console-oauth-config\") pod \"console-6878d7dcb-gd2n9\" (UID: \"85ee7f74-b251-4c68-b3af-0494b35df222\") " pod="openshift-console/console-6878d7dcb-gd2n9" Jul 30 19:26:13.242854 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.242780 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/85ee7f74-b251-4c68-b3af-0494b35df222-console-serving-cert\") pod \"console-6878d7dcb-gd2n9\" (UID: \"85ee7f74-b251-4c68-b3af-0494b35df222\") " pod="openshift-console/console-6878d7dcb-gd2n9" Jul 30 19:26:13.247076 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.247021 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjn9z\" (UniqueName: \"kubernetes.io/projected/85ee7f74-b251-4c68-b3af-0494b35df222-kube-api-access-sjn9z\") pod \"console-6878d7dcb-gd2n9\" (UID: \"85ee7f74-b251-4c68-b3af-0494b35df222\") " pod="openshift-console/console-6878d7dcb-gd2n9" Jul 30 19:26:13.419927 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.417725 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6878d7dcb-gd2n9" Jul 30 19:26:13.686842 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.686753 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 30 19:26:13.703800 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.703670 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6cbd87958-csgb5"] Jul 30 19:26:13.704149 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.703849 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.708019 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.707270 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 30 19:26:13.708019 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.707277 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 30 19:26:13.708019 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.707671 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 30 19:26:13.708019 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.707857 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 30 19:26:13.709530 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.709504 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 30 19:26:13.715213 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.714753 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 30 19:26:13.715213 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.714985 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7tgdm\"" Jul 30 19:26:13.716421 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.716326 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 30 19:26:13.716570 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.716553 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 30 19:26:13.716739 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.716718 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 30 19:26:13.716817 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.716637 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 30 19:26:13.718481 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.716928 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 30 19:26:13.718481 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.717219 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-e058oo53r4bao\"" Jul 30 19:26:13.718481 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.717368 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 30 19:26:13.718481 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.717779 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 30 19:26:13.718481 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.717976 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 30 19:26:13.720932 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.719934 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 30 19:26:13.720932 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.720813 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6878d7dcb-gd2n9"] Jul 30 19:26:13.724537 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.724347 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 30 19:26:13.740194 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.739357 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fae027fe-68b2-4024-9d0c-059513b1f746-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.740194 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.739423 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-web-config\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.740194 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.739462 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fae027fe-68b2-4024-9d0c-059513b1f746-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.740194 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.739502 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.740194 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.739526 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fae027fe-68b2-4024-9d0c-059513b1f746-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.740194 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.739551 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fae027fe-68b2-4024-9d0c-059513b1f746-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.740194 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.739587 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-config\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.740194 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.739622 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fae027fe-68b2-4024-9d0c-059513b1f746-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.740194 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.739647 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fae027fe-68b2-4024-9d0c-059513b1f746-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.740194 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.739676 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fae027fe-68b2-4024-9d0c-059513b1f746-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.740194 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.739701 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.740194 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.739725 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.740194 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.739748 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fae027fe-68b2-4024-9d0c-059513b1f746-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.740194 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.739777 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.740194 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.739809 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.740194 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.739841 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fae027fe-68b2-4024-9d0c-059513b1f746-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.741037 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.739875 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.741037 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.739896 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqcx7\" (UniqueName: \"kubernetes.io/projected/fae027fe-68b2-4024-9d0c-059513b1f746-kube-api-access-vqcx7\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.741037 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.739948 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fae027fe-68b2-4024-9d0c-059513b1f746-config-out\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.741037 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.739972 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.840834 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.840762 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fae027fe-68b2-4024-9d0c-059513b1f746-config-out\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.840834 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.840804 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.841305 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.840884 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fae027fe-68b2-4024-9d0c-059513b1f746-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.841305 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.840923 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-web-config\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.841305 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.840976 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fae027fe-68b2-4024-9d0c-059513b1f746-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.841305 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.841026 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.841305 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.841055 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fae027fe-68b2-4024-9d0c-059513b1f746-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.841305 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.841081 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fae027fe-68b2-4024-9d0c-059513b1f746-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.841305 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.841117 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-config\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.841305 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.841152 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fae027fe-68b2-4024-9d0c-059513b1f746-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.841305 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.841177 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fae027fe-68b2-4024-9d0c-059513b1f746-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.841305 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.841201 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fae027fe-68b2-4024-9d0c-059513b1f746-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.841305 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.841223 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.841305 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.841240 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.841305 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.841256 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fae027fe-68b2-4024-9d0c-059513b1f746-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.841305 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.841289 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.842015 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.841318 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.842015 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.841352 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fae027fe-68b2-4024-9d0c-059513b1f746-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.842015 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.841428 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.842015 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.841453 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqcx7\" (UniqueName: \"kubernetes.io/projected/fae027fe-68b2-4024-9d0c-059513b1f746-kube-api-access-vqcx7\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.842230 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.842207 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fae027fe-68b2-4024-9d0c-059513b1f746-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.842531 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.842483 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fae027fe-68b2-4024-9d0c-059513b1f746-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.842745 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.842721 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fae027fe-68b2-4024-9d0c-059513b1f746-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.842745 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.842739 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fae027fe-68b2-4024-9d0c-059513b1f746-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.843876 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.843669 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fae027fe-68b2-4024-9d0c-059513b1f746-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.845785 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.845754 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fae027fe-68b2-4024-9d0c-059513b1f746-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.845916 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.845895 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-config\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.848067 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.847103 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.848067 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.847199 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-web-config\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.848067 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.847238 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fae027fe-68b2-4024-9d0c-059513b1f746-config-out\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.848067 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.847423 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fae027fe-68b2-4024-9d0c-059513b1f746-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.848067 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.847636 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.848067 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.847946 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fae027fe-68b2-4024-9d0c-059513b1f746-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.848479 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.848104 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.848479 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.848337 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.848590 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.848513 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.848793 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.848767 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.848909 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.848889 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fae027fe-68b2-4024-9d0c-059513b1f746-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.849837 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.849812 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.856907 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:13.856884 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqcx7\" (UniqueName: \"kubernetes.io/projected/fae027fe-68b2-4024-9d0c-059513b1f746-kube-api-access-vqcx7\") pod \"prometheus-k8s-0\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:13.870695 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:26:13.870005 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc71030a2_fcdd_466c_8299_01c7f963c64d.slice/crio-2da43fd7a76f18d599496b1b8d438cb7dd50ec15efdfaa2ad5591ca78869995d WatchSource:0}: Error finding container 2da43fd7a76f18d599496b1b8d438cb7dd50ec15efdfaa2ad5591ca78869995d: Status 404 returned error can't find the container with id 2da43fd7a76f18d599496b1b8d438cb7dd50ec15efdfaa2ad5591ca78869995d Jul 30 19:26:13.871481 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:26:13.871319 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85ee7f74_b251_4c68_b3af_0494b35df222.slice/crio-c5a5743e631874b869df7138af7c5feb30d50c8857eb1d365d539f2173dc1277 WatchSource:0}: Error finding container c5a5743e631874b869df7138af7c5feb30d50c8857eb1d365d539f2173dc1277: Status 404 returned error can't find the container with id c5a5743e631874b869df7138af7c5feb30d50c8857eb1d365d539f2173dc1277 Jul 30 19:26:14.040399 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:14.037933 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:14.261531 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:14.260375 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 30 19:26:14.266247 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:26:14.266218 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfae027fe_68b2_4024_9d0c_059513b1f746.slice/crio-55f92846a17a1ccfddf84a0133844b8abd6f1638c6da0cfd5e3c4da6d9683ad8 WatchSource:0}: Error finding container 55f92846a17a1ccfddf84a0133844b8abd6f1638c6da0cfd5e3c4da6d9683ad8: Status 404 returned error can't find the container with id 55f92846a17a1ccfddf84a0133844b8abd6f1638c6da0cfd5e3c4da6d9683ad8 Jul 30 19:26:14.789101 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:14.789054 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fae027fe-68b2-4024-9d0c-059513b1f746","Type":"ContainerStarted","Data":"55f92846a17a1ccfddf84a0133844b8abd6f1638c6da0cfd5e3c4da6d9683ad8"} Jul 30 19:26:14.791573 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:14.791544 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-fd44f7995-8wknj" event={"ID":"b0261e45-1760-4a3e-925d-fe0d63dc7635","Type":"ContainerStarted","Data":"c3b0259dddb7063687161d198be79f73358b1b126f93e7ee1983405f2c701c01"} Jul 30 19:26:14.791714 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:14.791618 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-fd44f7995-8wknj" event={"ID":"b0261e45-1760-4a3e-925d-fe0d63dc7635","Type":"ContainerStarted","Data":"312f968cb25194b0929741cbc7e4385dd8566ec36b5c44763c844d2bbcd72ed9"} Jul 30 19:26:14.791714 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:14.791655 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-fd44f7995-8wknj" event={"ID":"b0261e45-1760-4a3e-925d-fe0d63dc7635","Type":"ContainerStarted","Data":"f89c6ad0e786ccf1947da8884c60b332f2415402f6c6fef853170a5b8acc3149"} Jul 30 19:26:14.793342 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:14.793318 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6878d7dcb-gd2n9" event={"ID":"85ee7f74-b251-4c68-b3af-0494b35df222","Type":"ContainerStarted","Data":"f4aec250384e6f1cb13f9030a0d66665bc26ed229902481d97a7d44ef67c8ac0"} Jul 30 19:26:14.793492 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:14.793357 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6878d7dcb-gd2n9" event={"ID":"85ee7f74-b251-4c68-b3af-0494b35df222","Type":"ContainerStarted","Data":"c5a5743e631874b869df7138af7c5feb30d50c8857eb1d365d539f2173dc1277"} Jul 30 19:26:14.794534 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:14.794511 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6cbd87958-csgb5" event={"ID":"c71030a2-fcdd-466c-8299-01c7f963c64d","Type":"ContainerStarted","Data":"2da43fd7a76f18d599496b1b8d438cb7dd50ec15efdfaa2ad5591ca78869995d"} Jul 30 19:26:14.813876 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:14.813821 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6878d7dcb-gd2n9" podStartSLOduration=1.813805636 podStartE2EDuration="1.813805636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 19:26:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:26:14.813678388 +0000 UTC m=+97.033237302" watchObservedRunningTime="2026-07-30 19:26:14.813805636 +0000 UTC m=+97.033364550" Jul 30 19:26:15.802155 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:15.802064 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-hpn6r" event={"ID":"ed0bae37-0a50-4f34-b0b8-bf912e2f75ed","Type":"ContainerStarted","Data":"843358a74411c03622555b291a3d6904f5551d0ba2d8094e9eb876106f4d0f9c"} Jul 30 19:26:15.821422 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:15.821343 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-hpn6r" podStartSLOduration=1.137737968 podStartE2EDuration="3.821323309s" totalImagesPullingTime="2.683585341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 19:26:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:26:15.819698588 +0000 UTC m=+98.039257502" watchObservedRunningTime="2026-07-30 19:26:15.821323309 +0000 UTC m=+98.040882222" Jul 30 19:26:16.807326 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:16.807288 2556 generic.go:356] "Generic (PLEG): container finished" podID="fae027fe-68b2-4024-9d0c-059513b1f746" containerID="25368a361c71b50d0ef24d1975df318ca76f3d08dd7ba79fca27c59409950a41" exitCode=0 Jul 30 19:26:16.807776 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:16.807391 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fae027fe-68b2-4024-9d0c-059513b1f746","Type":"ContainerDied","Data":"25368a361c71b50d0ef24d1975df318ca76f3d08dd7ba79fca27c59409950a41"} Jul 30 19:26:16.810717 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:16.810652 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-fd44f7995-8wknj" event={"ID":"b0261e45-1760-4a3e-925d-fe0d63dc7635","Type":"ContainerStarted","Data":"88207d6d09d7035f8fdf44f0339ba2d3637ceed294e0d59cd691151b184e9bb1"} Jul 30 19:26:16.810802 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:16.810730 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-fd44f7995-8wknj" event={"ID":"b0261e45-1760-4a3e-925d-fe0d63dc7635","Type":"ContainerStarted","Data":"a54b468fb6a561926710b09f0cf74d86d1819bc1f3cc7f8aa8e85b23cfbfa53b"} Jul 30 19:26:16.810982 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:16.810962 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-hpn6r" Jul 30 19:26:16.817354 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:16.817336 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-hpn6r" Jul 30 19:26:17.819853 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:17.819822 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-fd44f7995-8wknj" event={"ID":"b0261e45-1760-4a3e-925d-fe0d63dc7635","Type":"ContainerStarted","Data":"528468b116ae9d4add2f23dd8959b2076e7cd634c6156d1af5b2ed13ee475fc4"} Jul 30 19:26:17.820199 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:17.819962 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-fd44f7995-8wknj" Jul 30 19:26:17.821765 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:17.821702 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6cbd87958-csgb5" event={"ID":"c71030a2-fcdd-466c-8299-01c7f963c64d","Type":"ContainerStarted","Data":"b9e43c8f63401aeef8fe86664181ecd9a7f54178d9ca54311eb3cea4c30a558e"} Jul 30 19:26:17.844192 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:17.844135 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-fd44f7995-8wknj" podStartSLOduration=2.8484446180000003 podStartE2EDuration="7.844117025s" totalImagesPullingTime="4.995672407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 19:26:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:26:17.841833026 +0000 UTC m=+100.061391942" watchObservedRunningTime="2026-07-30 19:26:17.844117025 +0000 UTC m=+100.063675940" Jul 30 19:26:18.830510 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:18.830414 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6cbd87958-csgb5" event={"ID":"c71030a2-fcdd-466c-8299-01c7f963c64d","Type":"ContainerStarted","Data":"bbde7ee89a7d25e93fef8b2dce04f81f3d6ff31f772384702670a61382cc1a1a"} Jul 30 19:26:18.830510 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:18.830493 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6cbd87958-csgb5" event={"ID":"c71030a2-fcdd-466c-8299-01c7f963c64d","Type":"ContainerStarted","Data":"29a93070cc8cab9de501144b6c763d7dcb99d6a683d45eb2d5d6b742009f2072"} Jul 30 19:26:18.853054 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:18.853002 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6cbd87958-csgb5" podStartSLOduration=3.127091795 podStartE2EDuration="6.852985962s" totalImagesPullingTime="3.725894167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 19:26:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:26:18.850377914 +0000 UTC m=+101.069936829" watchObservedRunningTime="2026-07-30 19:26:18.852985962 +0000 UTC m=+101.072544874" Jul 30 19:26:18.935746 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:18.935710 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6878d7dcb-gd2n9"] Jul 30 19:26:18.974549 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:18.974515 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7d59785975-btdv4"] Jul 30 19:26:18.999675 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:18.999637 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d59785975-btdv4"] Jul 30 19:26:18.999826 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:18.999742 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d59785975-btdv4" Jul 30 19:26:19.091853 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:19.091796 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa53926f-3d57-4650-82ea-88f5ddacdb39-console-serving-cert\") pod \"console-7d59785975-btdv4\" (UID: \"aa53926f-3d57-4650-82ea-88f5ddacdb39\") " pod="openshift-console/console-7d59785975-btdv4" Jul 30 19:26:19.091853 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:19.091826 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aa53926f-3d57-4650-82ea-88f5ddacdb39-oauth-serving-cert\") pod \"console-7d59785975-btdv4\" (UID: \"aa53926f-3d57-4650-82ea-88f5ddacdb39\") " pod="openshift-console/console-7d59785975-btdv4" Jul 30 19:26:19.091853 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:19.091849 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa53926f-3d57-4650-82ea-88f5ddacdb39-trusted-ca-bundle\") pod \"console-7d59785975-btdv4\" (UID: \"aa53926f-3d57-4650-82ea-88f5ddacdb39\") " pod="openshift-console/console-7d59785975-btdv4" Jul 30 19:26:19.092029 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:19.091872 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aa53926f-3d57-4650-82ea-88f5ddacdb39-console-oauth-config\") pod \"console-7d59785975-btdv4\" (UID: \"aa53926f-3d57-4650-82ea-88f5ddacdb39\") " pod="openshift-console/console-7d59785975-btdv4" Jul 30 19:26:19.092029 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:19.091893 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fprr2\" (UniqueName: \"kubernetes.io/projected/aa53926f-3d57-4650-82ea-88f5ddacdb39-kube-api-access-fprr2\") pod \"console-7d59785975-btdv4\" (UID: \"aa53926f-3d57-4650-82ea-88f5ddacdb39\") " pod="openshift-console/console-7d59785975-btdv4" Jul 30 19:26:19.092029 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:19.091911 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa53926f-3d57-4650-82ea-88f5ddacdb39-service-ca\") pod \"console-7d59785975-btdv4\" (UID: \"aa53926f-3d57-4650-82ea-88f5ddacdb39\") " pod="openshift-console/console-7d59785975-btdv4" Jul 30 19:26:19.092029 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:19.091944 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aa53926f-3d57-4650-82ea-88f5ddacdb39-console-config\") pod \"console-7d59785975-btdv4\" (UID: \"aa53926f-3d57-4650-82ea-88f5ddacdb39\") " pod="openshift-console/console-7d59785975-btdv4" Jul 30 19:26:19.192432 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:19.192376 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa53926f-3d57-4650-82ea-88f5ddacdb39-trusted-ca-bundle\") pod \"console-7d59785975-btdv4\" (UID: \"aa53926f-3d57-4650-82ea-88f5ddacdb39\") " pod="openshift-console/console-7d59785975-btdv4" Jul 30 19:26:19.192578 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:19.192476 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aa53926f-3d57-4650-82ea-88f5ddacdb39-console-oauth-config\") pod \"console-7d59785975-btdv4\" (UID: \"aa53926f-3d57-4650-82ea-88f5ddacdb39\") " pod="openshift-console/console-7d59785975-btdv4" Jul 30 19:26:19.192578 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:19.192501 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fprr2\" (UniqueName: \"kubernetes.io/projected/aa53926f-3d57-4650-82ea-88f5ddacdb39-kube-api-access-fprr2\") pod \"console-7d59785975-btdv4\" (UID: \"aa53926f-3d57-4650-82ea-88f5ddacdb39\") " pod="openshift-console/console-7d59785975-btdv4" Jul 30 19:26:19.192578 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:19.192528 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa53926f-3d57-4650-82ea-88f5ddacdb39-service-ca\") pod \"console-7d59785975-btdv4\" (UID: \"aa53926f-3d57-4650-82ea-88f5ddacdb39\") " pod="openshift-console/console-7d59785975-btdv4" Jul 30 19:26:19.192731 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:19.192584 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aa53926f-3d57-4650-82ea-88f5ddacdb39-console-config\") pod \"console-7d59785975-btdv4\" (UID: \"aa53926f-3d57-4650-82ea-88f5ddacdb39\") " pod="openshift-console/console-7d59785975-btdv4" Jul 30 19:26:19.192731 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:19.192625 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa53926f-3d57-4650-82ea-88f5ddacdb39-console-serving-cert\") pod \"console-7d59785975-btdv4\" (UID: \"aa53926f-3d57-4650-82ea-88f5ddacdb39\") " pod="openshift-console/console-7d59785975-btdv4" Jul 30 19:26:19.192731 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:19.192647 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aa53926f-3d57-4650-82ea-88f5ddacdb39-oauth-serving-cert\") pod \"console-7d59785975-btdv4\" (UID: \"aa53926f-3d57-4650-82ea-88f5ddacdb39\") " pod="openshift-console/console-7d59785975-btdv4" Jul 30 19:26:19.193351 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:19.193319 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aa53926f-3d57-4650-82ea-88f5ddacdb39-console-config\") pod \"console-7d59785975-btdv4\" (UID: \"aa53926f-3d57-4650-82ea-88f5ddacdb39\") " pod="openshift-console/console-7d59785975-btdv4" Jul 30 19:26:19.193351 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:19.193335 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa53926f-3d57-4650-82ea-88f5ddacdb39-trusted-ca-bundle\") pod \"console-7d59785975-btdv4\" (UID: \"aa53926f-3d57-4650-82ea-88f5ddacdb39\") " pod="openshift-console/console-7d59785975-btdv4" Jul 30 19:26:19.193612 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:19.193344 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aa53926f-3d57-4650-82ea-88f5ddacdb39-oauth-serving-cert\") pod \"console-7d59785975-btdv4\" (UID: \"aa53926f-3d57-4650-82ea-88f5ddacdb39\") " pod="openshift-console/console-7d59785975-btdv4" Jul 30 19:26:19.193612 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:19.193441 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa53926f-3d57-4650-82ea-88f5ddacdb39-service-ca\") pod \"console-7d59785975-btdv4\" (UID: \"aa53926f-3d57-4650-82ea-88f5ddacdb39\") " pod="openshift-console/console-7d59785975-btdv4" Jul 30 19:26:19.195303 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:19.195276 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aa53926f-3d57-4650-82ea-88f5ddacdb39-console-oauth-config\") pod \"console-7d59785975-btdv4\" (UID: \"aa53926f-3d57-4650-82ea-88f5ddacdb39\") " pod="openshift-console/console-7d59785975-btdv4" Jul 30 19:26:19.195442 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:19.195362 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa53926f-3d57-4650-82ea-88f5ddacdb39-console-serving-cert\") pod \"console-7d59785975-btdv4\" (UID: \"aa53926f-3d57-4650-82ea-88f5ddacdb39\") " pod="openshift-console/console-7d59785975-btdv4" Jul 30 19:26:19.207085 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:19.207042 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fprr2\" (UniqueName: \"kubernetes.io/projected/aa53926f-3d57-4650-82ea-88f5ddacdb39-kube-api-access-fprr2\") pod \"console-7d59785975-btdv4\" (UID: \"aa53926f-3d57-4650-82ea-88f5ddacdb39\") " pod="openshift-console/console-7d59785975-btdv4" Jul 30 19:26:19.310735 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:19.310697 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d59785975-btdv4" Jul 30 19:26:20.488981 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:20.488956 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d59785975-btdv4"] Jul 30 19:26:20.491031 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:26:20.490993 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa53926f_3d57_4650_82ea_88f5ddacdb39.slice/crio-7bf1b557b386451e0062d2c738442d53f919b3547623e2e6607376adfe498610 WatchSource:0}: Error finding container 7bf1b557b386451e0062d2c738442d53f919b3547623e2e6607376adfe498610: Status 404 returned error can't find the container with id 7bf1b557b386451e0062d2c738442d53f919b3547623e2e6607376adfe498610 Jul 30 19:26:20.841414 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:20.841372 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d59785975-btdv4" event={"ID":"aa53926f-3d57-4650-82ea-88f5ddacdb39","Type":"ContainerStarted","Data":"dfec189d0e787f843f35b55d5e828b8f1c8d24f00ca7c0d74dff259a2a832290"} Jul 30 19:26:20.841507 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:20.841437 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d59785975-btdv4" event={"ID":"aa53926f-3d57-4650-82ea-88f5ddacdb39","Type":"ContainerStarted","Data":"7bf1b557b386451e0062d2c738442d53f919b3547623e2e6607376adfe498610"} Jul 30 19:26:20.862652 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:20.862608 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7d59785975-btdv4" podStartSLOduration=2.862594465 podStartE2EDuration="2.862594465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 19:26:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:26:20.861132235 +0000 UTC m=+103.080691149" watchObservedRunningTime="2026-07-30 19:26:20.862594465 +0000 UTC m=+103.082153378" Jul 30 19:26:21.848680 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:21.848642 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fae027fe-68b2-4024-9d0c-059513b1f746","Type":"ContainerStarted","Data":"a139fd384252469ef2d01ad5d22bf127037166befc97b89c979c31412c38e193"} Jul 30 19:26:21.849090 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:21.848758 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fae027fe-68b2-4024-9d0c-059513b1f746","Type":"ContainerStarted","Data":"06b246917cf0177a0828ad280f5e3058d8fe8069d89039bd2d83526eb6aa45c4"} Jul 30 19:26:21.849090 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:21.848800 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fae027fe-68b2-4024-9d0c-059513b1f746","Type":"ContainerStarted","Data":"db1d52bfa5b064ea643698289b577327b1030a52482280de0ef7398054eb5ce2"} Jul 30 19:26:21.849090 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:21.848834 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fae027fe-68b2-4024-9d0c-059513b1f746","Type":"ContainerStarted","Data":"f6b4194afa57b0ca408335b45319bd15c49faa6f39fba72713737605844a77c2"} Jul 30 19:26:21.849090 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:21.848872 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fae027fe-68b2-4024-9d0c-059513b1f746","Type":"ContainerStarted","Data":"4ce96aa17360ef36e4bc645be93be461afab96e3524a3cd70ce1ebcbad79f711"} Jul 30 19:26:21.849090 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:21.848907 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fae027fe-68b2-4024-9d0c-059513b1f746","Type":"ContainerStarted","Data":"f0b3068de40f29e5d71b8cf96b0eeb9fd1a8b8afa61e3a1a8cdaabe8bb3545f0"} Jul 30 19:26:21.878565 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:21.877870 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.716544882 podStartE2EDuration="8.877850128s" totalImagesPullingTime="6.132346152s" totalInitContainerRuntime="28.959094ms" isStatefulPod=false podCreationTimestamp="2026-07-30 19:26:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:26:21.876028308 +0000 UTC m=+104.095587237" watchObservedRunningTime="2026-07-30 19:26:21.877850128 +0000 UTC m=+104.097409042" Jul 30 19:26:23.418269 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:23.418240 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6878d7dcb-gd2n9" Jul 30 19:26:23.837123 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:23.837053 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-fd44f7995-8wknj" Jul 30 19:26:24.039306 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:24.039276 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:26:24.678337 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:24.678304 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-czm5g" Jul 30 19:26:29.311798 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:29.311756 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7d59785975-btdv4" Jul 30 19:26:29.312278 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:29.311842 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7d59785975-btdv4" Jul 30 19:26:29.316792 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:29.316766 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7d59785975-btdv4" Jul 30 19:26:29.885370 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:29.885338 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7d59785975-btdv4" Jul 30 19:26:37.150840 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:37.150798 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-84fc9cc987-hm25f" podUID="1e3d5ab2-a94c-46e8-8347-baa3755abb91" containerName="console" containerID="cri-o://60d54341d14fd8703789d8e57196a7f40a835a1d58b22be6ddc640fc9431cb41" gracePeriod=15 Jul 30 19:26:37.439805 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:37.439783 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84fc9cc987-hm25f" Jul 30 19:26:37.540708 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:37.540681 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1e3d5ab2-a94c-46e8-8347-baa3755abb91-console-config\") pod \"1e3d5ab2-a94c-46e8-8347-baa3755abb91\" (UID: \"1e3d5ab2-a94c-46e8-8347-baa3755abb91\") " Jul 30 19:26:37.540847 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:37.540746 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1e3d5ab2-a94c-46e8-8347-baa3755abb91-console-oauth-config\") pod \"1e3d5ab2-a94c-46e8-8347-baa3755abb91\" (UID: \"1e3d5ab2-a94c-46e8-8347-baa3755abb91\") " Jul 30 19:26:37.540847 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:37.540769 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1e3d5ab2-a94c-46e8-8347-baa3755abb91-oauth-serving-cert\") pod \"1e3d5ab2-a94c-46e8-8347-baa3755abb91\" (UID: \"1e3d5ab2-a94c-46e8-8347-baa3755abb91\") " Jul 30 19:26:37.540847 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:37.540812 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zbpmw\" (UniqueName: \"kubernetes.io/projected/1e3d5ab2-a94c-46e8-8347-baa3755abb91-kube-api-access-zbpmw\") pod \"1e3d5ab2-a94c-46e8-8347-baa3755abb91\" (UID: \"1e3d5ab2-a94c-46e8-8347-baa3755abb91\") " Jul 30 19:26:37.540847 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:37.540830 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e3d5ab2-a94c-46e8-8347-baa3755abb91-console-serving-cert\") pod \"1e3d5ab2-a94c-46e8-8347-baa3755abb91\" (UID: \"1e3d5ab2-a94c-46e8-8347-baa3755abb91\") " Jul 30 19:26:37.541058 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:37.540862 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1e3d5ab2-a94c-46e8-8347-baa3755abb91-service-ca\") pod \"1e3d5ab2-a94c-46e8-8347-baa3755abb91\" (UID: \"1e3d5ab2-a94c-46e8-8347-baa3755abb91\") " Jul 30 19:26:37.541197 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:37.541166 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e3d5ab2-a94c-46e8-8347-baa3755abb91-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1e3d5ab2-a94c-46e8-8347-baa3755abb91" (UID: "1e3d5ab2-a94c-46e8-8347-baa3755abb91"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 19:26:37.541197 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:37.541182 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e3d5ab2-a94c-46e8-8347-baa3755abb91-console-config" (OuterVolumeSpecName: "console-config") pod "1e3d5ab2-a94c-46e8-8347-baa3755abb91" (UID: "1e3d5ab2-a94c-46e8-8347-baa3755abb91"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 19:26:37.541329 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:37.541243 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e3d5ab2-a94c-46e8-8347-baa3755abb91-service-ca" (OuterVolumeSpecName: "service-ca") pod "1e3d5ab2-a94c-46e8-8347-baa3755abb91" (UID: "1e3d5ab2-a94c-46e8-8347-baa3755abb91"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 19:26:37.543087 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:37.543065 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e3d5ab2-a94c-46e8-8347-baa3755abb91-kube-api-access-zbpmw" (OuterVolumeSpecName: "kube-api-access-zbpmw") pod "1e3d5ab2-a94c-46e8-8347-baa3755abb91" (UID: "1e3d5ab2-a94c-46e8-8347-baa3755abb91"). InnerVolumeSpecName "kube-api-access-zbpmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 19:26:37.543323 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:37.543307 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e3d5ab2-a94c-46e8-8347-baa3755abb91-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1e3d5ab2-a94c-46e8-8347-baa3755abb91" (UID: "1e3d5ab2-a94c-46e8-8347-baa3755abb91"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 19:26:37.543446 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:37.543424 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e3d5ab2-a94c-46e8-8347-baa3755abb91-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1e3d5ab2-a94c-46e8-8347-baa3755abb91" (UID: "1e3d5ab2-a94c-46e8-8347-baa3755abb91"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 19:26:37.641437 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:37.641411 2556 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1e3d5ab2-a94c-46e8-8347-baa3755abb91-console-oauth-config\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:26:37.641437 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:37.641432 2556 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1e3d5ab2-a94c-46e8-8347-baa3755abb91-oauth-serving-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:26:37.641556 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:37.641442 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbpmw\" (UniqueName: \"kubernetes.io/projected/1e3d5ab2-a94c-46e8-8347-baa3755abb91-kube-api-access-zbpmw\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:26:37.641556 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:37.641450 2556 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e3d5ab2-a94c-46e8-8347-baa3755abb91-console-serving-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:26:37.641556 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:37.641459 2556 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1e3d5ab2-a94c-46e8-8347-baa3755abb91-service-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:26:37.641556 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:37.641467 2556 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1e3d5ab2-a94c-46e8-8347-baa3755abb91-console-config\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:26:37.915933 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:37.915908 2556 generic.go:356] "Generic (PLEG): container finished" podID="1e3d5ab2-a94c-46e8-8347-baa3755abb91" containerID="60d54341d14fd8703789d8e57196a7f40a835a1d58b22be6ddc640fc9431cb41" exitCode=0 Jul 30 19:26:37.916045 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:37.915953 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84fc9cc987-hm25f" Jul 30 19:26:37.916105 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:37.915956 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84fc9cc987-hm25f" event={"ID":"1e3d5ab2-a94c-46e8-8347-baa3755abb91","Type":"ContainerDied","Data":"60d54341d14fd8703789d8e57196a7f40a835a1d58b22be6ddc640fc9431cb41"} Jul 30 19:26:37.916161 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:37.916099 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84fc9cc987-hm25f" event={"ID":"1e3d5ab2-a94c-46e8-8347-baa3755abb91","Type":"ContainerDied","Data":"2b9ab9bb49f12909c09ed3bdd1e5c2f91a7c6b9a90c480e06424521d6655365b"} Jul 30 19:26:37.916161 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:37.916141 2556 scope.go:117] "RemoveContainer" containerID="60d54341d14fd8703789d8e57196a7f40a835a1d58b22be6ddc640fc9431cb41" Jul 30 19:26:37.925213 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:37.925197 2556 scope.go:117] "RemoveContainer" containerID="60d54341d14fd8703789d8e57196a7f40a835a1d58b22be6ddc640fc9431cb41" Jul 30 19:26:37.925496 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:26:37.925475 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60d54341d14fd8703789d8e57196a7f40a835a1d58b22be6ddc640fc9431cb41\": container with ID starting with 60d54341d14fd8703789d8e57196a7f40a835a1d58b22be6ddc640fc9431cb41 not found: ID does not exist" containerID="60d54341d14fd8703789d8e57196a7f40a835a1d58b22be6ddc640fc9431cb41" Jul 30 19:26:37.925567 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:37.925511 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60d54341d14fd8703789d8e57196a7f40a835a1d58b22be6ddc640fc9431cb41"} err="failed to get container status \"60d54341d14fd8703789d8e57196a7f40a835a1d58b22be6ddc640fc9431cb41\": rpc error: code = NotFound desc = could not find container \"60d54341d14fd8703789d8e57196a7f40a835a1d58b22be6ddc640fc9431cb41\": container with ID starting with 60d54341d14fd8703789d8e57196a7f40a835a1d58b22be6ddc640fc9431cb41 not found: ID does not exist" Jul 30 19:26:37.936348 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:37.936328 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84fc9cc987-hm25f"] Jul 30 19:26:37.940007 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:37.939988 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-84fc9cc987-hm25f"] Jul 30 19:26:38.323130 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:38.323100 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e3d5ab2-a94c-46e8-8347-baa3755abb91" path="/var/lib/kubelet/pods/1e3d5ab2-a94c-46e8-8347-baa3755abb91/volumes" Jul 30 19:26:43.957596 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:43.957556 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6878d7dcb-gd2n9" podUID="85ee7f74-b251-4c68-b3af-0494b35df222" containerName="console" containerID="cri-o://f4aec250384e6f1cb13f9030a0d66665bc26ed229902481d97a7d44ef67c8ac0" gracePeriod=15 Jul 30 19:26:44.244638 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:44.244566 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6878d7dcb-gd2n9" Jul 30 19:26:44.396525 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:44.396476 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/85ee7f74-b251-4c68-b3af-0494b35df222-console-oauth-config\") pod \"85ee7f74-b251-4c68-b3af-0494b35df222\" (UID: \"85ee7f74-b251-4c68-b3af-0494b35df222\") " Jul 30 19:26:44.396706 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:44.396569 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/85ee7f74-b251-4c68-b3af-0494b35df222-console-serving-cert\") pod \"85ee7f74-b251-4c68-b3af-0494b35df222\" (UID: \"85ee7f74-b251-4c68-b3af-0494b35df222\") " Jul 30 19:26:44.396706 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:44.396600 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/85ee7f74-b251-4c68-b3af-0494b35df222-console-config\") pod \"85ee7f74-b251-4c68-b3af-0494b35df222\" (UID: \"85ee7f74-b251-4c68-b3af-0494b35df222\") " Jul 30 19:26:44.396706 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:44.396625 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/85ee7f74-b251-4c68-b3af-0494b35df222-service-ca\") pod \"85ee7f74-b251-4c68-b3af-0494b35df222\" (UID: \"85ee7f74-b251-4c68-b3af-0494b35df222\") " Jul 30 19:26:44.396706 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:44.396652 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/85ee7f74-b251-4c68-b3af-0494b35df222-oauth-serving-cert\") pod \"85ee7f74-b251-4c68-b3af-0494b35df222\" (UID: \"85ee7f74-b251-4c68-b3af-0494b35df222\") " Jul 30 19:26:44.396706 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:44.396695 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85ee7f74-b251-4c68-b3af-0494b35df222-trusted-ca-bundle\") pod \"85ee7f74-b251-4c68-b3af-0494b35df222\" (UID: \"85ee7f74-b251-4c68-b3af-0494b35df222\") " Jul 30 19:26:44.396949 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:44.396726 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sjn9z\" (UniqueName: \"kubernetes.io/projected/85ee7f74-b251-4c68-b3af-0494b35df222-kube-api-access-sjn9z\") pod \"85ee7f74-b251-4c68-b3af-0494b35df222\" (UID: \"85ee7f74-b251-4c68-b3af-0494b35df222\") " Jul 30 19:26:44.397082 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:44.397043 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85ee7f74-b251-4c68-b3af-0494b35df222-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "85ee7f74-b251-4c68-b3af-0494b35df222" (UID: "85ee7f74-b251-4c68-b3af-0494b35df222"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 19:26:44.397164 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:44.397005 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85ee7f74-b251-4c68-b3af-0494b35df222-console-config" (OuterVolumeSpecName: "console-config") pod "85ee7f74-b251-4c68-b3af-0494b35df222" (UID: "85ee7f74-b251-4c68-b3af-0494b35df222"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 19:26:44.397237 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:44.397212 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85ee7f74-b251-4c68-b3af-0494b35df222-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "85ee7f74-b251-4c68-b3af-0494b35df222" (UID: "85ee7f74-b251-4c68-b3af-0494b35df222"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 19:26:44.397406 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:44.397366 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85ee7f74-b251-4c68-b3af-0494b35df222-service-ca" (OuterVolumeSpecName: "service-ca") pod "85ee7f74-b251-4c68-b3af-0494b35df222" (UID: "85ee7f74-b251-4c68-b3af-0494b35df222"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 19:26:44.398769 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:44.398736 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85ee7f74-b251-4c68-b3af-0494b35df222-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "85ee7f74-b251-4c68-b3af-0494b35df222" (UID: "85ee7f74-b251-4c68-b3af-0494b35df222"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 19:26:44.398855 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:44.398828 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85ee7f74-b251-4c68-b3af-0494b35df222-kube-api-access-sjn9z" (OuterVolumeSpecName: "kube-api-access-sjn9z") pod "85ee7f74-b251-4c68-b3af-0494b35df222" (UID: "85ee7f74-b251-4c68-b3af-0494b35df222"). InnerVolumeSpecName "kube-api-access-sjn9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 19:26:44.398855 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:44.398838 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85ee7f74-b251-4c68-b3af-0494b35df222-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "85ee7f74-b251-4c68-b3af-0494b35df222" (UID: "85ee7f74-b251-4c68-b3af-0494b35df222"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 19:26:44.498122 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:44.498040 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjn9z\" (UniqueName: \"kubernetes.io/projected/85ee7f74-b251-4c68-b3af-0494b35df222-kube-api-access-sjn9z\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:26:44.498122 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:44.498078 2556 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/85ee7f74-b251-4c68-b3af-0494b35df222-console-oauth-config\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:26:44.498122 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:44.498093 2556 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/85ee7f74-b251-4c68-b3af-0494b35df222-console-serving-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:26:44.498122 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:44.498108 2556 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/85ee7f74-b251-4c68-b3af-0494b35df222-console-config\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:26:44.498122 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:44.498118 2556 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/85ee7f74-b251-4c68-b3af-0494b35df222-service-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:26:44.498122 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:44.498126 2556 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/85ee7f74-b251-4c68-b3af-0494b35df222-oauth-serving-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:26:44.498431 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:44.498134 2556 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85ee7f74-b251-4c68-b3af-0494b35df222-trusted-ca-bundle\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:26:44.944618 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:44.944585 2556 generic.go:356] "Generic (PLEG): container finished" podID="85ee7f74-b251-4c68-b3af-0494b35df222" containerID="f4aec250384e6f1cb13f9030a0d66665bc26ed229902481d97a7d44ef67c8ac0" exitCode=0 Jul 30 19:26:44.944771 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:44.944643 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6878d7dcb-gd2n9" Jul 30 19:26:44.944771 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:44.944668 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6878d7dcb-gd2n9" event={"ID":"85ee7f74-b251-4c68-b3af-0494b35df222","Type":"ContainerDied","Data":"f4aec250384e6f1cb13f9030a0d66665bc26ed229902481d97a7d44ef67c8ac0"} Jul 30 19:26:44.944771 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:44.944725 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6878d7dcb-gd2n9" event={"ID":"85ee7f74-b251-4c68-b3af-0494b35df222","Type":"ContainerDied","Data":"c5a5743e631874b869df7138af7c5feb30d50c8857eb1d365d539f2173dc1277"} Jul 30 19:26:44.944771 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:44.944746 2556 scope.go:117] "RemoveContainer" containerID="f4aec250384e6f1cb13f9030a0d66665bc26ed229902481d97a7d44ef67c8ac0" Jul 30 19:26:44.953355 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:44.953339 2556 scope.go:117] "RemoveContainer" containerID="f4aec250384e6f1cb13f9030a0d66665bc26ed229902481d97a7d44ef67c8ac0" Jul 30 19:26:44.953658 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:26:44.953637 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4aec250384e6f1cb13f9030a0d66665bc26ed229902481d97a7d44ef67c8ac0\": container with ID starting with f4aec250384e6f1cb13f9030a0d66665bc26ed229902481d97a7d44ef67c8ac0 not found: ID does not exist" containerID="f4aec250384e6f1cb13f9030a0d66665bc26ed229902481d97a7d44ef67c8ac0" Jul 30 19:26:44.953715 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:44.953669 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4aec250384e6f1cb13f9030a0d66665bc26ed229902481d97a7d44ef67c8ac0"} err="failed to get container status \"f4aec250384e6f1cb13f9030a0d66665bc26ed229902481d97a7d44ef67c8ac0\": rpc error: code = NotFound desc = could not find container \"f4aec250384e6f1cb13f9030a0d66665bc26ed229902481d97a7d44ef67c8ac0\": container with ID starting with f4aec250384e6f1cb13f9030a0d66665bc26ed229902481d97a7d44ef67c8ac0 not found: ID does not exist" Jul 30 19:26:44.965417 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:44.965393 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6878d7dcb-gd2n9"] Jul 30 19:26:44.968754 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:44.968734 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6878d7dcb-gd2n9"] Jul 30 19:26:46.323636 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:26:46.323600 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85ee7f74-b251-4c68-b3af-0494b35df222" path="/var/lib/kubelet/pods/85ee7f74-b251-4c68-b3af-0494b35df222/volumes" Jul 30 19:27:14.039133 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:14.039085 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:14.059541 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:14.059518 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:14.079203 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:14.079181 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:31.974614 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:31.974526 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 30 19:27:31.975248 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:31.974916 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fae027fe-68b2-4024-9d0c-059513b1f746" containerName="prometheus" containerID="cri-o://f0b3068de40f29e5d71b8cf96b0eeb9fd1a8b8afa61e3a1a8cdaabe8bb3545f0" gracePeriod=600 Jul 30 19:27:31.975248 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:31.974940 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fae027fe-68b2-4024-9d0c-059513b1f746" containerName="kube-rbac-proxy-web" containerID="cri-o://db1d52bfa5b064ea643698289b577327b1030a52482280de0ef7398054eb5ce2" gracePeriod=600 Jul 30 19:27:31.975248 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:31.974965 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fae027fe-68b2-4024-9d0c-059513b1f746" containerName="thanos-sidecar" containerID="cri-o://f6b4194afa57b0ca408335b45319bd15c49faa6f39fba72713737605844a77c2" gracePeriod=600 Jul 30 19:27:31.975248 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:31.974978 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fae027fe-68b2-4024-9d0c-059513b1f746" containerName="config-reloader" containerID="cri-o://4ce96aa17360ef36e4bc645be93be461afab96e3524a3cd70ce1ebcbad79f711" gracePeriod=600 Jul 30 19:27:31.975248 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:31.974956 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fae027fe-68b2-4024-9d0c-059513b1f746" containerName="kube-rbac-proxy" containerID="cri-o://06b246917cf0177a0828ad280f5e3058d8fe8069d89039bd2d83526eb6aa45c4" gracePeriod=600 Jul 30 19:27:31.975248 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:31.974921 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fae027fe-68b2-4024-9d0c-059513b1f746" containerName="kube-rbac-proxy-thanos" containerID="cri-o://a139fd384252469ef2d01ad5d22bf127037166befc97b89c979c31412c38e193" gracePeriod=600 Jul 30 19:27:32.138116 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.138078 2556 generic.go:356] "Generic (PLEG): container finished" podID="fae027fe-68b2-4024-9d0c-059513b1f746" containerID="a139fd384252469ef2d01ad5d22bf127037166befc97b89c979c31412c38e193" exitCode=0 Jul 30 19:27:32.138116 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.138105 2556 generic.go:356] "Generic (PLEG): container finished" podID="fae027fe-68b2-4024-9d0c-059513b1f746" containerID="06b246917cf0177a0828ad280f5e3058d8fe8069d89039bd2d83526eb6aa45c4" exitCode=0 Jul 30 19:27:32.138116 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.138114 2556 generic.go:356] "Generic (PLEG): container finished" podID="fae027fe-68b2-4024-9d0c-059513b1f746" containerID="db1d52bfa5b064ea643698289b577327b1030a52482280de0ef7398054eb5ce2" exitCode=0 Jul 30 19:27:32.138116 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.138123 2556 generic.go:356] "Generic (PLEG): container finished" podID="fae027fe-68b2-4024-9d0c-059513b1f746" containerID="f6b4194afa57b0ca408335b45319bd15c49faa6f39fba72713737605844a77c2" exitCode=0 Jul 30 19:27:32.138367 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.138129 2556 generic.go:356] "Generic (PLEG): container finished" podID="fae027fe-68b2-4024-9d0c-059513b1f746" containerID="4ce96aa17360ef36e4bc645be93be461afab96e3524a3cd70ce1ebcbad79f711" exitCode=0 Jul 30 19:27:32.138367 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.138135 2556 generic.go:356] "Generic (PLEG): container finished" podID="fae027fe-68b2-4024-9d0c-059513b1f746" containerID="f0b3068de40f29e5d71b8cf96b0eeb9fd1a8b8afa61e3a1a8cdaabe8bb3545f0" exitCode=0 Jul 30 19:27:32.138367 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.138152 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fae027fe-68b2-4024-9d0c-059513b1f746","Type":"ContainerDied","Data":"a139fd384252469ef2d01ad5d22bf127037166befc97b89c979c31412c38e193"} Jul 30 19:27:32.138367 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.138292 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fae027fe-68b2-4024-9d0c-059513b1f746","Type":"ContainerDied","Data":"06b246917cf0177a0828ad280f5e3058d8fe8069d89039bd2d83526eb6aa45c4"} Jul 30 19:27:32.138367 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.138339 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fae027fe-68b2-4024-9d0c-059513b1f746","Type":"ContainerDied","Data":"db1d52bfa5b064ea643698289b577327b1030a52482280de0ef7398054eb5ce2"} Jul 30 19:27:32.138543 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.138412 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fae027fe-68b2-4024-9d0c-059513b1f746","Type":"ContainerDied","Data":"f6b4194afa57b0ca408335b45319bd15c49faa6f39fba72713737605844a77c2"} Jul 30 19:27:32.138543 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.138449 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fae027fe-68b2-4024-9d0c-059513b1f746","Type":"ContainerDied","Data":"4ce96aa17360ef36e4bc645be93be461afab96e3524a3cd70ce1ebcbad79f711"} Jul 30 19:27:32.138543 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.138484 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fae027fe-68b2-4024-9d0c-059513b1f746","Type":"ContainerDied","Data":"f0b3068de40f29e5d71b8cf96b0eeb9fd1a8b8afa61e3a1a8cdaabe8bb3545f0"} Jul 30 19:27:32.226310 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.226250 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:32.270903 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.270478 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"fae027fe-68b2-4024-9d0c-059513b1f746\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " Jul 30 19:27:32.270903 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.270526 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fae027fe-68b2-4024-9d0c-059513b1f746-prometheus-k8s-rulefiles-2\") pod \"fae027fe-68b2-4024-9d0c-059513b1f746\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " Jul 30 19:27:32.270903 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.270565 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-config\") pod \"fae027fe-68b2-4024-9d0c-059513b1f746\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " Jul 30 19:27:32.271242 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.271007 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"fae027fe-68b2-4024-9d0c-059513b1f746\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " Jul 30 19:27:32.271242 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.271042 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fae027fe-68b2-4024-9d0c-059513b1f746-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "fae027fe-68b2-4024-9d0c-059513b1f746" (UID: "fae027fe-68b2-4024-9d0c-059513b1f746"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 19:27:32.271242 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.271067 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-thanos-prometheus-http-client-file\") pod \"fae027fe-68b2-4024-9d0c-059513b1f746\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " Jul 30 19:27:32.271242 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.271102 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fae027fe-68b2-4024-9d0c-059513b1f746-configmap-kubelet-serving-ca-bundle\") pod \"fae027fe-68b2-4024-9d0c-059513b1f746\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " Jul 30 19:27:32.271242 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.271130 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-secret-metrics-client-certs\") pod \"fae027fe-68b2-4024-9d0c-059513b1f746\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " Jul 30 19:27:32.271242 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.271177 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fae027fe-68b2-4024-9d0c-059513b1f746-prometheus-k8s-rulefiles-0\") pod \"fae027fe-68b2-4024-9d0c-059513b1f746\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " Jul 30 19:27:32.271242 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.271203 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-secret-grpc-tls\") pod \"fae027fe-68b2-4024-9d0c-059513b1f746\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " Jul 30 19:27:32.271242 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.271237 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fae027fe-68b2-4024-9d0c-059513b1f746-prometheus-k8s-db\") pod \"fae027fe-68b2-4024-9d0c-059513b1f746\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " Jul 30 19:27:32.271677 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.271265 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-web-config\") pod \"fae027fe-68b2-4024-9d0c-059513b1f746\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " Jul 30 19:27:32.271677 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.271291 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-secret-kube-rbac-proxy\") pod \"fae027fe-68b2-4024-9d0c-059513b1f746\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " Jul 30 19:27:32.271677 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.271322 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fae027fe-68b2-4024-9d0c-059513b1f746-prometheus-trusted-ca-bundle\") pod \"fae027fe-68b2-4024-9d0c-059513b1f746\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " Jul 30 19:27:32.271677 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.271357 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fae027fe-68b2-4024-9d0c-059513b1f746-configmap-serving-certs-ca-bundle\") pod \"fae027fe-68b2-4024-9d0c-059513b1f746\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " Jul 30 19:27:32.271677 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.271447 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fae027fe-68b2-4024-9d0c-059513b1f746-prometheus-k8s-rulefiles-1\") pod \"fae027fe-68b2-4024-9d0c-059513b1f746\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " Jul 30 19:27:32.271677 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.271476 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vqcx7\" (UniqueName: \"kubernetes.io/projected/fae027fe-68b2-4024-9d0c-059513b1f746-kube-api-access-vqcx7\") pod \"fae027fe-68b2-4024-9d0c-059513b1f746\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " Jul 30 19:27:32.271677 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.271518 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-secret-prometheus-k8s-tls\") pod \"fae027fe-68b2-4024-9d0c-059513b1f746\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " Jul 30 19:27:32.271677 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.271568 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fae027fe-68b2-4024-9d0c-059513b1f746-tls-assets\") pod \"fae027fe-68b2-4024-9d0c-059513b1f746\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " Jul 30 19:27:32.271677 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.271618 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fae027fe-68b2-4024-9d0c-059513b1f746-configmap-metrics-client-ca\") pod \"fae027fe-68b2-4024-9d0c-059513b1f746\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " Jul 30 19:27:32.271677 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.271645 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fae027fe-68b2-4024-9d0c-059513b1f746-config-out\") pod \"fae027fe-68b2-4024-9d0c-059513b1f746\" (UID: \"fae027fe-68b2-4024-9d0c-059513b1f746\") " Jul 30 19:27:32.272180 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.271934 2556 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fae027fe-68b2-4024-9d0c-059513b1f746-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:27:32.272930 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.272424 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fae027fe-68b2-4024-9d0c-059513b1f746-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "fae027fe-68b2-4024-9d0c-059513b1f746" (UID: "fae027fe-68b2-4024-9d0c-059513b1f746"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 19:27:32.272930 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.272727 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fae027fe-68b2-4024-9d0c-059513b1f746-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "fae027fe-68b2-4024-9d0c-059513b1f746" (UID: "fae027fe-68b2-4024-9d0c-059513b1f746"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 19:27:32.273144 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.273070 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fae027fe-68b2-4024-9d0c-059513b1f746-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "fae027fe-68b2-4024-9d0c-059513b1f746" (UID: "fae027fe-68b2-4024-9d0c-059513b1f746"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 19:27:32.273222 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.273198 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fae027fe-68b2-4024-9d0c-059513b1f746-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "fae027fe-68b2-4024-9d0c-059513b1f746" (UID: "fae027fe-68b2-4024-9d0c-059513b1f746"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 19:27:32.274177 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.273840 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fae027fe-68b2-4024-9d0c-059513b1f746-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "fae027fe-68b2-4024-9d0c-059513b1f746" (UID: "fae027fe-68b2-4024-9d0c-059513b1f746"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 19:27:32.274177 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.274155 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fae027fe-68b2-4024-9d0c-059513b1f746-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "fae027fe-68b2-4024-9d0c-059513b1f746" (UID: "fae027fe-68b2-4024-9d0c-059513b1f746"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 19:27:32.274598 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.274562 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fae027fe-68b2-4024-9d0c-059513b1f746-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "fae027fe-68b2-4024-9d0c-059513b1f746" (UID: "fae027fe-68b2-4024-9d0c-059513b1f746"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 19:27:32.278149 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.278102 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "fae027fe-68b2-4024-9d0c-059513b1f746" (UID: "fae027fe-68b2-4024-9d0c-059513b1f746"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 19:27:32.280557 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.280530 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fae027fe-68b2-4024-9d0c-059513b1f746-kube-api-access-vqcx7" (OuterVolumeSpecName: "kube-api-access-vqcx7") pod "fae027fe-68b2-4024-9d0c-059513b1f746" (UID: "fae027fe-68b2-4024-9d0c-059513b1f746"). InnerVolumeSpecName "kube-api-access-vqcx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 19:27:32.280660 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.280617 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-config" (OuterVolumeSpecName: "config") pod "fae027fe-68b2-4024-9d0c-059513b1f746" (UID: "fae027fe-68b2-4024-9d0c-059513b1f746"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 19:27:32.280732 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.280648 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "fae027fe-68b2-4024-9d0c-059513b1f746" (UID: "fae027fe-68b2-4024-9d0c-059513b1f746"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 19:27:32.280732 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.280706 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "fae027fe-68b2-4024-9d0c-059513b1f746" (UID: "fae027fe-68b2-4024-9d0c-059513b1f746"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 19:27:32.280732 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.280726 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "fae027fe-68b2-4024-9d0c-059513b1f746" (UID: "fae027fe-68b2-4024-9d0c-059513b1f746"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 19:27:32.280891 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.280764 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "fae027fe-68b2-4024-9d0c-059513b1f746" (UID: "fae027fe-68b2-4024-9d0c-059513b1f746"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 19:27:32.281043 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.281008 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "fae027fe-68b2-4024-9d0c-059513b1f746" (UID: "fae027fe-68b2-4024-9d0c-059513b1f746"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 19:27:32.281113 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.281037 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "fae027fe-68b2-4024-9d0c-059513b1f746" (UID: "fae027fe-68b2-4024-9d0c-059513b1f746"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 19:27:32.281704 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.281551 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fae027fe-68b2-4024-9d0c-059513b1f746-config-out" (OuterVolumeSpecName: "config-out") pod "fae027fe-68b2-4024-9d0c-059513b1f746" (UID: "fae027fe-68b2-4024-9d0c-059513b1f746"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 19:27:32.281845 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.281817 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fae027fe-68b2-4024-9d0c-059513b1f746-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "fae027fe-68b2-4024-9d0c-059513b1f746" (UID: "fae027fe-68b2-4024-9d0c-059513b1f746"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 19:27:32.293978 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.293940 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-web-config" (OuterVolumeSpecName: "web-config") pod "fae027fe-68b2-4024-9d0c-059513b1f746" (UID: "fae027fe-68b2-4024-9d0c-059513b1f746"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 19:27:32.373363 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.373269 2556 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fae027fe-68b2-4024-9d0c-059513b1f746-configmap-metrics-client-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:27:32.373363 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.373304 2556 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fae027fe-68b2-4024-9d0c-059513b1f746-config-out\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:27:32.373363 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.373322 2556 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:27:32.373363 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.373350 2556 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-config\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:27:32.373671 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.373378 2556 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:27:32.373671 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.373424 2556 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:27:32.373671 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.373442 2556 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fae027fe-68b2-4024-9d0c-059513b1f746-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:27:32.373671 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.373458 2556 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-secret-metrics-client-certs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:27:32.373671 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.373472 2556 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fae027fe-68b2-4024-9d0c-059513b1f746-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:27:32.373671 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.373484 2556 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-secret-grpc-tls\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:27:32.373671 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.373494 2556 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fae027fe-68b2-4024-9d0c-059513b1f746-prometheus-k8s-db\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:27:32.373671 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.373502 2556 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-web-config\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:27:32.373671 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.373511 2556 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-secret-kube-rbac-proxy\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:27:32.373671 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.373519 2556 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fae027fe-68b2-4024-9d0c-059513b1f746-prometheus-trusted-ca-bundle\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:27:32.373671 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.373527 2556 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fae027fe-68b2-4024-9d0c-059513b1f746-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:27:32.373671 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.373553 2556 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fae027fe-68b2-4024-9d0c-059513b1f746-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:27:32.373671 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.373572 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqcx7\" (UniqueName: \"kubernetes.io/projected/fae027fe-68b2-4024-9d0c-059513b1f746-kube-api-access-vqcx7\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:27:32.373671 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.373589 2556 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fae027fe-68b2-4024-9d0c-059513b1f746-secret-prometheus-k8s-tls\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:27:32.373671 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:32.373604 2556 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fae027fe-68b2-4024-9d0c-059513b1f746-tls-assets\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:27:33.144640 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.144604 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fae027fe-68b2-4024-9d0c-059513b1f746","Type":"ContainerDied","Data":"55f92846a17a1ccfddf84a0133844b8abd6f1638c6da0cfd5e3c4da6d9683ad8"} Jul 30 19:27:33.145044 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.144685 2556 scope.go:117] "RemoveContainer" containerID="a139fd384252469ef2d01ad5d22bf127037166befc97b89c979c31412c38e193" Jul 30 19:27:33.145044 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.144614 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.153072 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.153055 2556 scope.go:117] "RemoveContainer" containerID="06b246917cf0177a0828ad280f5e3058d8fe8069d89039bd2d83526eb6aa45c4" Jul 30 19:27:33.159871 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.159856 2556 scope.go:117] "RemoveContainer" containerID="db1d52bfa5b064ea643698289b577327b1030a52482280de0ef7398054eb5ce2" Jul 30 19:27:33.167485 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.167463 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 30 19:27:33.167621 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.167609 2556 scope.go:117] "RemoveContainer" containerID="f6b4194afa57b0ca408335b45319bd15c49faa6f39fba72713737605844a77c2" Jul 30 19:27:33.172181 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.172156 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 30 19:27:33.174912 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.174894 2556 scope.go:117] "RemoveContainer" containerID="4ce96aa17360ef36e4bc645be93be461afab96e3524a3cd70ce1ebcbad79f711" Jul 30 19:27:33.181390 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.181361 2556 scope.go:117] "RemoveContainer" containerID="f0b3068de40f29e5d71b8cf96b0eeb9fd1a8b8afa61e3a1a8cdaabe8bb3545f0" Jul 30 19:27:33.188580 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.188562 2556 scope.go:117] "RemoveContainer" containerID="25368a361c71b50d0ef24d1975df318ca76f3d08dd7ba79fca27c59409950a41" Jul 30 19:27:33.198133 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.198113 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 30 19:27:33.198493 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.198479 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fae027fe-68b2-4024-9d0c-059513b1f746" containerName="config-reloader" Jul 30 19:27:33.198544 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.198494 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="fae027fe-68b2-4024-9d0c-059513b1f746" containerName="config-reloader" Jul 30 19:27:33.198544 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.198501 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fae027fe-68b2-4024-9d0c-059513b1f746" containerName="kube-rbac-proxy" Jul 30 19:27:33.198544 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.198506 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="fae027fe-68b2-4024-9d0c-059513b1f746" containerName="kube-rbac-proxy" Jul 30 19:27:33.198544 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.198513 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e3d5ab2-a94c-46e8-8347-baa3755abb91" containerName="console" Jul 30 19:27:33.198544 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.198518 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e3d5ab2-a94c-46e8-8347-baa3755abb91" containerName="console" Jul 30 19:27:33.198544 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.198526 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fae027fe-68b2-4024-9d0c-059513b1f746" containerName="init-config-reloader" Jul 30 19:27:33.198544 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.198531 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="fae027fe-68b2-4024-9d0c-059513b1f746" containerName="init-config-reloader" Jul 30 19:27:33.198544 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.198539 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85ee7f74-b251-4c68-b3af-0494b35df222" containerName="console" Jul 30 19:27:33.198544 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.198543 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="85ee7f74-b251-4c68-b3af-0494b35df222" containerName="console" Jul 30 19:27:33.198826 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.198553 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fae027fe-68b2-4024-9d0c-059513b1f746" containerName="thanos-sidecar" Jul 30 19:27:33.198826 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.198559 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="fae027fe-68b2-4024-9d0c-059513b1f746" containerName="thanos-sidecar" Jul 30 19:27:33.198826 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.198569 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fae027fe-68b2-4024-9d0c-059513b1f746" containerName="prometheus" Jul 30 19:27:33.198826 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.198574 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="fae027fe-68b2-4024-9d0c-059513b1f746" containerName="prometheus" Jul 30 19:27:33.198826 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.198581 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fae027fe-68b2-4024-9d0c-059513b1f746" containerName="kube-rbac-proxy-web" Jul 30 19:27:33.198826 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.198586 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="fae027fe-68b2-4024-9d0c-059513b1f746" containerName="kube-rbac-proxy-web" Jul 30 19:27:33.198826 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.198594 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fae027fe-68b2-4024-9d0c-059513b1f746" containerName="kube-rbac-proxy-thanos" Jul 30 19:27:33.198826 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.198599 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="fae027fe-68b2-4024-9d0c-059513b1f746" containerName="kube-rbac-proxy-thanos" Jul 30 19:27:33.198826 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.198645 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="fae027fe-68b2-4024-9d0c-059513b1f746" containerName="kube-rbac-proxy" Jul 30 19:27:33.198826 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.198654 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="85ee7f74-b251-4c68-b3af-0494b35df222" containerName="console" Jul 30 19:27:33.198826 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.198660 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="fae027fe-68b2-4024-9d0c-059513b1f746" containerName="kube-rbac-proxy-thanos" Jul 30 19:27:33.198826 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.198666 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="fae027fe-68b2-4024-9d0c-059513b1f746" containerName="kube-rbac-proxy-web" Jul 30 19:27:33.198826 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.198671 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e3d5ab2-a94c-46e8-8347-baa3755abb91" containerName="console" Jul 30 19:27:33.198826 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.198676 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="fae027fe-68b2-4024-9d0c-059513b1f746" containerName="config-reloader" Jul 30 19:27:33.198826 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.198681 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="fae027fe-68b2-4024-9d0c-059513b1f746" containerName="thanos-sidecar" Jul 30 19:27:33.198826 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.198686 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="fae027fe-68b2-4024-9d0c-059513b1f746" containerName="prometheus" Jul 30 19:27:33.202547 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.202528 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.205581 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.205559 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 30 19:27:33.205677 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.205597 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 30 19:27:33.205677 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.205620 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 30 19:27:33.205677 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.205620 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 30 19:27:33.205677 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.205665 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 30 19:27:33.205861 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.205673 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 30 19:27:33.205961 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.205946 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 30 19:27:33.206017 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.205951 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7tgdm\"" Jul 30 19:27:33.206055 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.206027 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 30 19:27:33.206096 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.205989 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 30 19:27:33.206613 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.206596 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-e058oo53r4bao\"" Jul 30 19:27:33.206693 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.206616 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 30 19:27:33.206693 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.206621 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 30 19:27:33.206795 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.206618 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 30 19:27:33.206795 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.206768 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 30 19:27:33.209617 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.209599 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 30 19:27:33.212088 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.212067 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 30 19:27:33.221028 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.221009 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 30 19:27:33.282153 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.282066 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fece5390-a1e9-4126-86bd-91af2af5ccf7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.282153 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.282104 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fece5390-a1e9-4126-86bd-91af2af5ccf7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.282153 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.282133 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fece5390-a1e9-4126-86bd-91af2af5ccf7-config\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.282432 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.282161 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fece5390-a1e9-4126-86bd-91af2af5ccf7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.282432 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.282185 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fece5390-a1e9-4126-86bd-91af2af5ccf7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.282432 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.282210 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fece5390-a1e9-4126-86bd-91af2af5ccf7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.282432 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.282255 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fece5390-a1e9-4126-86bd-91af2af5ccf7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.282432 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.282279 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fece5390-a1e9-4126-86bd-91af2af5ccf7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.282432 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.282299 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fece5390-a1e9-4126-86bd-91af2af5ccf7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.282432 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.282324 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptrb6\" (UniqueName: \"kubernetes.io/projected/fece5390-a1e9-4126-86bd-91af2af5ccf7-kube-api-access-ptrb6\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.282432 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.282350 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fece5390-a1e9-4126-86bd-91af2af5ccf7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.282432 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.282372 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fece5390-a1e9-4126-86bd-91af2af5ccf7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.282432 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.282434 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fece5390-a1e9-4126-86bd-91af2af5ccf7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.282793 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.282462 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fece5390-a1e9-4126-86bd-91af2af5ccf7-config-out\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.282793 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.282490 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fece5390-a1e9-4126-86bd-91af2af5ccf7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.282793 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.282546 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fece5390-a1e9-4126-86bd-91af2af5ccf7-web-config\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.282793 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.282599 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fece5390-a1e9-4126-86bd-91af2af5ccf7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.282793 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.282619 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fece5390-a1e9-4126-86bd-91af2af5ccf7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.282793 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.282640 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fece5390-a1e9-4126-86bd-91af2af5ccf7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.282793 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.282658 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fece5390-a1e9-4126-86bd-91af2af5ccf7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.384042 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.383999 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fece5390-a1e9-4126-86bd-91af2af5ccf7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.384042 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.384045 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fece5390-a1e9-4126-86bd-91af2af5ccf7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.384208 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.384095 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fece5390-a1e9-4126-86bd-91af2af5ccf7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.384208 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.384133 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fece5390-a1e9-4126-86bd-91af2af5ccf7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.384208 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.384162 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fece5390-a1e9-4126-86bd-91af2af5ccf7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.384208 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.384184 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fece5390-a1e9-4126-86bd-91af2af5ccf7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.384428 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.384213 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptrb6\" (UniqueName: \"kubernetes.io/projected/fece5390-a1e9-4126-86bd-91af2af5ccf7-kube-api-access-ptrb6\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.384428 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.384266 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fece5390-a1e9-4126-86bd-91af2af5ccf7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.384428 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.384313 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fece5390-a1e9-4126-86bd-91af2af5ccf7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.384428 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.384413 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fece5390-a1e9-4126-86bd-91af2af5ccf7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.384656 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.384451 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fece5390-a1e9-4126-86bd-91af2af5ccf7-config-out\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.384656 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.384482 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fece5390-a1e9-4126-86bd-91af2af5ccf7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.384656 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.384513 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fece5390-a1e9-4126-86bd-91af2af5ccf7-web-config\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.384656 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.384560 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fece5390-a1e9-4126-86bd-91af2af5ccf7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.384656 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.384586 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fece5390-a1e9-4126-86bd-91af2af5ccf7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.384656 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.384612 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fece5390-a1e9-4126-86bd-91af2af5ccf7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.384656 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.384620 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fece5390-a1e9-4126-86bd-91af2af5ccf7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.385080 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.384679 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fece5390-a1e9-4126-86bd-91af2af5ccf7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.385080 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.384716 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fece5390-a1e9-4126-86bd-91af2af5ccf7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.385080 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.384726 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fece5390-a1e9-4126-86bd-91af2af5ccf7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.385080 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.384767 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fece5390-a1e9-4126-86bd-91af2af5ccf7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.385080 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.384804 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fece5390-a1e9-4126-86bd-91af2af5ccf7-config\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.385337 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.385302 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fece5390-a1e9-4126-86bd-91af2af5ccf7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.387243 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.387109 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fece5390-a1e9-4126-86bd-91af2af5ccf7-config-out\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.387243 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.387166 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fece5390-a1e9-4126-86bd-91af2af5ccf7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.387243 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.387185 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fece5390-a1e9-4126-86bd-91af2af5ccf7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.387941 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.387528 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fece5390-a1e9-4126-86bd-91af2af5ccf7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.387941 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.387565 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fece5390-a1e9-4126-86bd-91af2af5ccf7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.388580 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.388544 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fece5390-a1e9-4126-86bd-91af2af5ccf7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.389141 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.389021 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fece5390-a1e9-4126-86bd-91af2af5ccf7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.389141 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.389086 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fece5390-a1e9-4126-86bd-91af2af5ccf7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.389829 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.389662 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fece5390-a1e9-4126-86bd-91af2af5ccf7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.389829 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.389726 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/fece5390-a1e9-4126-86bd-91af2af5ccf7-config\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.390525 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.390052 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fece5390-a1e9-4126-86bd-91af2af5ccf7-web-config\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.390525 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.390472 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fece5390-a1e9-4126-86bd-91af2af5ccf7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.392245 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.390619 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fece5390-a1e9-4126-86bd-91af2af5ccf7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.392245 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.391074 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fece5390-a1e9-4126-86bd-91af2af5ccf7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.392594 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.392570 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fece5390-a1e9-4126-86bd-91af2af5ccf7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.392893 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.392867 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fece5390-a1e9-4126-86bd-91af2af5ccf7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.393376 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.393355 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptrb6\" (UniqueName: \"kubernetes.io/projected/fece5390-a1e9-4126-86bd-91af2af5ccf7-kube-api-access-ptrb6\") pod \"prometheus-k8s-0\" (UID: \"fece5390-a1e9-4126-86bd-91af2af5ccf7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.512538 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.512503 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:33.647429 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:33.647400 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 30 19:27:33.649286 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:27:33.649262 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfece5390_a1e9_4126_86bd_91af2af5ccf7.slice/crio-027d70a3f731ea9a99c0ead915069863b89266f0eae5c147d1b28315018e1a07 WatchSource:0}: Error finding container 027d70a3f731ea9a99c0ead915069863b89266f0eae5c147d1b28315018e1a07: Status 404 returned error can't find the container with id 027d70a3f731ea9a99c0ead915069863b89266f0eae5c147d1b28315018e1a07 Jul 30 19:27:34.150252 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:34.150216 2556 generic.go:356] "Generic (PLEG): container finished" podID="fece5390-a1e9-4126-86bd-91af2af5ccf7" containerID="003f6e33145b881f0949744d7505b36393a655773b38c2c5b573675d72f31bd0" exitCode=0 Jul 30 19:27:34.150689 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:34.150296 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fece5390-a1e9-4126-86bd-91af2af5ccf7","Type":"ContainerDied","Data":"003f6e33145b881f0949744d7505b36393a655773b38c2c5b573675d72f31bd0"} Jul 30 19:27:34.150689 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:34.150376 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fece5390-a1e9-4126-86bd-91af2af5ccf7","Type":"ContainerStarted","Data":"027d70a3f731ea9a99c0ead915069863b89266f0eae5c147d1b28315018e1a07"} Jul 30 19:27:34.326216 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:34.326179 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fae027fe-68b2-4024-9d0c-059513b1f746" path="/var/lib/kubelet/pods/fae027fe-68b2-4024-9d0c-059513b1f746/volumes" Jul 30 19:27:35.158897 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:35.158863 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fece5390-a1e9-4126-86bd-91af2af5ccf7","Type":"ContainerStarted","Data":"803e0515a54f85e88144352dc2a089c3b9a4149b20e7672cf8a186ac2bf1dc3a"} Jul 30 19:27:35.159405 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:35.158936 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fece5390-a1e9-4126-86bd-91af2af5ccf7","Type":"ContainerStarted","Data":"828a5428f2f7ed694701165d8d288de5a42b101c1228060783bc0d812674064f"} Jul 30 19:27:35.159405 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:35.158980 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fece5390-a1e9-4126-86bd-91af2af5ccf7","Type":"ContainerStarted","Data":"084ce6f246e33da6716d8f484296d870f5cb3cf3e864373f4a94dce0a08735e2"} Jul 30 19:27:35.159405 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:35.159020 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fece5390-a1e9-4126-86bd-91af2af5ccf7","Type":"ContainerStarted","Data":"8a6f969a75e1c52bc261be599e73393e0da8103bbfc5fbf273cf0d6574437163"} Jul 30 19:27:35.159405 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:35.159069 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fece5390-a1e9-4126-86bd-91af2af5ccf7","Type":"ContainerStarted","Data":"a651c8167758f832d25690072ba104820c699b45450ceff870481a70eccaee1c"} Jul 30 19:27:35.159405 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:35.159105 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fece5390-a1e9-4126-86bd-91af2af5ccf7","Type":"ContainerStarted","Data":"8a8fbe25f23667071892d022d436c8cc9fdc81395017757486a31d7425489cfd"} Jul 30 19:27:35.187025 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:35.186969 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.164729072 podStartE2EDuration="2.18695301s" totalImagesPullingTime="0s" totalInitContainerRuntime="22.223938ms" isStatefulPod=false podCreationTimestamp="2026-07-30 19:27:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:27:35.185001938 +0000 UTC m=+177.404560853" watchObservedRunningTime="2026-07-30 19:27:35.18695301 +0000 UTC m=+177.406511923" Jul 30 19:27:38.512942 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:38.512862 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:27:52.551186 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:27:52.551140 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d59785975-btdv4"] Jul 30 19:28:17.570954 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:28:17.570889 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7d59785975-btdv4" podUID="aa53926f-3d57-4650-82ea-88f5ddacdb39" containerName="console" containerID="cri-o://dfec189d0e787f843f35b55d5e828b8f1c8d24f00ca7c0d74dff259a2a832290" gracePeriod=15 Jul 30 19:28:17.813921 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:28:17.813895 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d59785975-btdv4" Jul 30 19:28:17.879206 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:28:17.879179 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa53926f-3d57-4650-82ea-88f5ddacdb39-service-ca\") pod \"aa53926f-3d57-4650-82ea-88f5ddacdb39\" (UID: \"aa53926f-3d57-4650-82ea-88f5ddacdb39\") " Jul 30 19:28:17.879418 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:28:17.879241 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fprr2\" (UniqueName: \"kubernetes.io/projected/aa53926f-3d57-4650-82ea-88f5ddacdb39-kube-api-access-fprr2\") pod \"aa53926f-3d57-4650-82ea-88f5ddacdb39\" (UID: \"aa53926f-3d57-4650-82ea-88f5ddacdb39\") " Jul 30 19:28:17.879418 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:28:17.879259 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aa53926f-3d57-4650-82ea-88f5ddacdb39-console-config\") pod \"aa53926f-3d57-4650-82ea-88f5ddacdb39\" (UID: \"aa53926f-3d57-4650-82ea-88f5ddacdb39\") " Jul 30 19:28:17.879418 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:28:17.879282 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aa53926f-3d57-4650-82ea-88f5ddacdb39-console-oauth-config\") pod \"aa53926f-3d57-4650-82ea-88f5ddacdb39\" (UID: \"aa53926f-3d57-4650-82ea-88f5ddacdb39\") " Jul 30 19:28:17.879418 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:28:17.879307 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa53926f-3d57-4650-82ea-88f5ddacdb39-trusted-ca-bundle\") pod \"aa53926f-3d57-4650-82ea-88f5ddacdb39\" (UID: \"aa53926f-3d57-4650-82ea-88f5ddacdb39\") " Jul 30 19:28:17.879418 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:28:17.879340 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa53926f-3d57-4650-82ea-88f5ddacdb39-console-serving-cert\") pod \"aa53926f-3d57-4650-82ea-88f5ddacdb39\" (UID: \"aa53926f-3d57-4650-82ea-88f5ddacdb39\") " Jul 30 19:28:17.879418 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:28:17.879356 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aa53926f-3d57-4650-82ea-88f5ddacdb39-oauth-serving-cert\") pod \"aa53926f-3d57-4650-82ea-88f5ddacdb39\" (UID: \"aa53926f-3d57-4650-82ea-88f5ddacdb39\") " Jul 30 19:28:17.879786 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:28:17.879753 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa53926f-3d57-4650-82ea-88f5ddacdb39-service-ca" (OuterVolumeSpecName: "service-ca") pod "aa53926f-3d57-4650-82ea-88f5ddacdb39" (UID: "aa53926f-3d57-4650-82ea-88f5ddacdb39"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 19:28:17.879924 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:28:17.879899 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa53926f-3d57-4650-82ea-88f5ddacdb39-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "aa53926f-3d57-4650-82ea-88f5ddacdb39" (UID: "aa53926f-3d57-4650-82ea-88f5ddacdb39"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 19:28:17.880013 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:28:17.879977 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa53926f-3d57-4650-82ea-88f5ddacdb39-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "aa53926f-3d57-4650-82ea-88f5ddacdb39" (UID: "aa53926f-3d57-4650-82ea-88f5ddacdb39"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 19:28:17.880177 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:28:17.880157 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa53926f-3d57-4650-82ea-88f5ddacdb39-console-config" (OuterVolumeSpecName: "console-config") pod "aa53926f-3d57-4650-82ea-88f5ddacdb39" (UID: "aa53926f-3d57-4650-82ea-88f5ddacdb39"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 19:28:17.881541 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:28:17.881514 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa53926f-3d57-4650-82ea-88f5ddacdb39-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "aa53926f-3d57-4650-82ea-88f5ddacdb39" (UID: "aa53926f-3d57-4650-82ea-88f5ddacdb39"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 19:28:17.881622 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:28:17.881545 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa53926f-3d57-4650-82ea-88f5ddacdb39-kube-api-access-fprr2" (OuterVolumeSpecName: "kube-api-access-fprr2") pod "aa53926f-3d57-4650-82ea-88f5ddacdb39" (UID: "aa53926f-3d57-4650-82ea-88f5ddacdb39"). InnerVolumeSpecName "kube-api-access-fprr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 19:28:17.881622 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:28:17.881598 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa53926f-3d57-4650-82ea-88f5ddacdb39-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "aa53926f-3d57-4650-82ea-88f5ddacdb39" (UID: "aa53926f-3d57-4650-82ea-88f5ddacdb39"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 19:28:17.980639 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:28:17.980598 2556 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa53926f-3d57-4650-82ea-88f5ddacdb39-service-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:28:17.980639 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:28:17.980632 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fprr2\" (UniqueName: \"kubernetes.io/projected/aa53926f-3d57-4650-82ea-88f5ddacdb39-kube-api-access-fprr2\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:28:17.980639 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:28:17.980647 2556 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aa53926f-3d57-4650-82ea-88f5ddacdb39-console-config\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:28:17.980892 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:28:17.980659 2556 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aa53926f-3d57-4650-82ea-88f5ddacdb39-console-oauth-config\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:28:17.980892 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:28:17.980671 2556 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa53926f-3d57-4650-82ea-88f5ddacdb39-trusted-ca-bundle\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:28:17.980892 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:28:17.980682 2556 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa53926f-3d57-4650-82ea-88f5ddacdb39-console-serving-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:28:17.980892 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:28:17.980693 2556 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aa53926f-3d57-4650-82ea-88f5ddacdb39-oauth-serving-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:28:18.332106 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:28:18.332075 2556 generic.go:356] "Generic (PLEG): container finished" podID="aa53926f-3d57-4650-82ea-88f5ddacdb39" containerID="dfec189d0e787f843f35b55d5e828b8f1c8d24f00ca7c0d74dff259a2a832290" exitCode=0 Jul 30 19:28:18.332248 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:28:18.332115 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d59785975-btdv4" Jul 30 19:28:18.332248 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:28:18.332145 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d59785975-btdv4" event={"ID":"aa53926f-3d57-4650-82ea-88f5ddacdb39","Type":"ContainerDied","Data":"dfec189d0e787f843f35b55d5e828b8f1c8d24f00ca7c0d74dff259a2a832290"} Jul 30 19:28:18.332248 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:28:18.332190 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d59785975-btdv4" event={"ID":"aa53926f-3d57-4650-82ea-88f5ddacdb39","Type":"ContainerDied","Data":"7bf1b557b386451e0062d2c738442d53f919b3547623e2e6607376adfe498610"} Jul 30 19:28:18.332248 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:28:18.332218 2556 scope.go:117] "RemoveContainer" containerID="dfec189d0e787f843f35b55d5e828b8f1c8d24f00ca7c0d74dff259a2a832290" Jul 30 19:28:18.340365 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:28:18.340349 2556 scope.go:117] "RemoveContainer" containerID="dfec189d0e787f843f35b55d5e828b8f1c8d24f00ca7c0d74dff259a2a832290" Jul 30 19:28:18.340629 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:28:18.340610 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfec189d0e787f843f35b55d5e828b8f1c8d24f00ca7c0d74dff259a2a832290\": container with ID starting with dfec189d0e787f843f35b55d5e828b8f1c8d24f00ca7c0d74dff259a2a832290 not found: ID does not exist" containerID="dfec189d0e787f843f35b55d5e828b8f1c8d24f00ca7c0d74dff259a2a832290" Jul 30 19:28:18.340675 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:28:18.340639 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfec189d0e787f843f35b55d5e828b8f1c8d24f00ca7c0d74dff259a2a832290"} err="failed to get container status \"dfec189d0e787f843f35b55d5e828b8f1c8d24f00ca7c0d74dff259a2a832290\": rpc error: code = NotFound desc = could not find container \"dfec189d0e787f843f35b55d5e828b8f1c8d24f00ca7c0d74dff259a2a832290\": container with ID starting with dfec189d0e787f843f35b55d5e828b8f1c8d24f00ca7c0d74dff259a2a832290 not found: ID does not exist" Jul 30 19:28:18.354119 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:28:18.354090 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d59785975-btdv4"] Jul 30 19:28:18.357696 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:28:18.357677 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7d59785975-btdv4"] Jul 30 19:28:20.322857 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:28:20.322825 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa53926f-3d57-4650-82ea-88f5ddacdb39" path="/var/lib/kubelet/pods/aa53926f-3d57-4650-82ea-88f5ddacdb39/volumes" Jul 30 19:28:33.513323 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:28:33.513287 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:28:33.528631 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:28:33.528596 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:28:34.407729 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:28:34.407699 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 30 19:29:33.213474 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:29:33.213432 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ssx6d"] Jul 30 19:29:33.214059 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:29:33.213983 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa53926f-3d57-4650-82ea-88f5ddacdb39" containerName="console" Jul 30 19:29:33.214059 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:29:33.214003 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa53926f-3d57-4650-82ea-88f5ddacdb39" containerName="console" Jul 30 19:29:33.214186 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:29:33.214074 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa53926f-3d57-4650-82ea-88f5ddacdb39" containerName="console" Jul 30 19:29:33.216211 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:29:33.216191 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ssx6d" Jul 30 19:29:33.218749 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:29:33.218718 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 30 19:29:33.218881 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:29:33.218826 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-d8n4d\"" Jul 30 19:29:33.219552 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:29:33.219537 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 30 19:29:33.226362 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:29:33.226339 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ssx6d"] Jul 30 19:29:33.327785 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:29:33.327748 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ae14bcf-646c-47b0-868a-36fb4b775e8a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-ssx6d\" (UID: \"3ae14bcf-646c-47b0-868a-36fb4b775e8a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ssx6d" Jul 30 19:29:33.327965 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:29:33.327826 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx4w4\" (UniqueName: \"kubernetes.io/projected/3ae14bcf-646c-47b0-868a-36fb4b775e8a-kube-api-access-fx4w4\") pod \"openshift-lws-operator-fd8ccff4c-ssx6d\" (UID: \"3ae14bcf-646c-47b0-868a-36fb4b775e8a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ssx6d" Jul 30 19:29:33.428731 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:29:33.428689 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx4w4\" (UniqueName: \"kubernetes.io/projected/3ae14bcf-646c-47b0-868a-36fb4b775e8a-kube-api-access-fx4w4\") pod \"openshift-lws-operator-fd8ccff4c-ssx6d\" (UID: \"3ae14bcf-646c-47b0-868a-36fb4b775e8a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ssx6d" Jul 30 19:29:33.428916 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:29:33.428741 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ae14bcf-646c-47b0-868a-36fb4b775e8a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-ssx6d\" (UID: \"3ae14bcf-646c-47b0-868a-36fb4b775e8a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ssx6d" Jul 30 19:29:33.429172 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:29:33.429148 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ae14bcf-646c-47b0-868a-36fb4b775e8a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-ssx6d\" (UID: \"3ae14bcf-646c-47b0-868a-36fb4b775e8a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ssx6d" Jul 30 19:29:33.437091 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:29:33.437061 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx4w4\" (UniqueName: \"kubernetes.io/projected/3ae14bcf-646c-47b0-868a-36fb4b775e8a-kube-api-access-fx4w4\") pod \"openshift-lws-operator-fd8ccff4c-ssx6d\" (UID: \"3ae14bcf-646c-47b0-868a-36fb4b775e8a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ssx6d" Jul 30 19:29:33.534839 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:29:33.534757 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ssx6d" Jul 30 19:29:33.665692 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:29:33.665661 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ssx6d"] Jul 30 19:29:33.668279 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:29:33.668248 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ae14bcf_646c_47b0_868a_36fb4b775e8a.slice/crio-dcc9680b073f6e4766ee07141fff68ae38b99c328228e2a30bf224c99402d6ca WatchSource:0}: Error finding container dcc9680b073f6e4766ee07141fff68ae38b99c328228e2a30bf224c99402d6ca: Status 404 returned error can't find the container with id dcc9680b073f6e4766ee07141fff68ae38b99c328228e2a30bf224c99402d6ca Jul 30 19:29:34.634401 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:29:34.634341 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ssx6d" event={"ID":"3ae14bcf-646c-47b0-868a-36fb4b775e8a","Type":"ContainerStarted","Data":"dcc9680b073f6e4766ee07141fff68ae38b99c328228e2a30bf224c99402d6ca"} Jul 30 19:29:37.650290 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:29:37.650251 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ssx6d" event={"ID":"3ae14bcf-646c-47b0-868a-36fb4b775e8a","Type":"ContainerStarted","Data":"2cd4ec699afc85c5f3b648afa146ea13cf654a97c9bb3172032f1db78c3207c6"} Jul 30 19:29:37.667745 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:29:37.667676 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ssx6d" podStartSLOduration=1.092073927 podStartE2EDuration="4.667654057s" totalImagesPullingTime="3.57558013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 19:29:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:29:37.666122657 +0000 UTC m=+299.885681572" watchObservedRunningTime="2026-07-30 19:29:37.667654057 +0000 UTC m=+299.887212983" Jul 30 19:29:38.174930 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:29:38.174893 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-795qw_4cdd383f-6493-4d83-80fa-f4ce17e8416e/ovn-acl-logging/0.log" Jul 30 19:29:38.177227 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:29:38.177200 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-795qw_4cdd383f-6493-4d83-80fa-f4ce17e8416e/ovn-acl-logging/0.log" Jul 30 19:29:38.181554 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:29:38.181531 2556 kubelet.go:1648] "Image garbage collection succeeded" Jul 30 19:29:55.071911 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:29:55.071875 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-76fdfbdbf4-w5r56"] Jul 30 19:29:55.075156 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:29:55.075139 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-76fdfbdbf4-w5r56" Jul 30 19:29:55.078737 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:29:55.078714 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 30 19:29:55.078874 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:29:55.078744 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 30 19:29:55.078874 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:29:55.078748 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 30 19:29:55.079130 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:29:55.079116 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-29w77\"" Jul 30 19:29:55.079199 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:29:55.079178 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 30 19:29:55.108425 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:29:55.108393 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-76fdfbdbf4-w5r56"] Jul 30 19:29:55.116200 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:29:55.116166 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0604020a-98cf-4d6c-8ee7-a12cbda5f73d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76fdfbdbf4-w5r56\" (UID: \"0604020a-98cf-4d6c-8ee7-a12cbda5f73d\") " pod="opendatahub/opendatahub-operator-controller-manager-76fdfbdbf4-w5r56" Jul 30 19:29:55.116341 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:29:55.116220 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0604020a-98cf-4d6c-8ee7-a12cbda5f73d-webhook-cert\") pod \"opendatahub-operator-controller-manager-76fdfbdbf4-w5r56\" (UID: \"0604020a-98cf-4d6c-8ee7-a12cbda5f73d\") " pod="opendatahub/opendatahub-operator-controller-manager-76fdfbdbf4-w5r56" Jul 30 19:29:55.116341 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:29:55.116238 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0604020a-98cf-4d6c-8ee7-a12cbda5f73d-manifests\") pod \"opendatahub-operator-controller-manager-76fdfbdbf4-w5r56\" (UID: \"0604020a-98cf-4d6c-8ee7-a12cbda5f73d\") " pod="opendatahub/opendatahub-operator-controller-manager-76fdfbdbf4-w5r56" Jul 30 19:29:55.116341 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:29:55.116322 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0604020a-98cf-4d6c-8ee7-a12cbda5f73d-tmp\") pod \"opendatahub-operator-controller-manager-76fdfbdbf4-w5r56\" (UID: \"0604020a-98cf-4d6c-8ee7-a12cbda5f73d\") " pod="opendatahub/opendatahub-operator-controller-manager-76fdfbdbf4-w5r56" Jul 30 19:29:55.116458 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:29:55.116352 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzfmq\" (UniqueName: \"kubernetes.io/projected/0604020a-98cf-4d6c-8ee7-a12cbda5f73d-kube-api-access-xzfmq\") pod \"opendatahub-operator-controller-manager-76fdfbdbf4-w5r56\" (UID: \"0604020a-98cf-4d6c-8ee7-a12cbda5f73d\") " pod="opendatahub/opendatahub-operator-controller-manager-76fdfbdbf4-w5r56" Jul 30 19:29:55.217785 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:29:55.217744 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0604020a-98cf-4d6c-8ee7-a12cbda5f73d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76fdfbdbf4-w5r56\" (UID: \"0604020a-98cf-4d6c-8ee7-a12cbda5f73d\") " pod="opendatahub/opendatahub-operator-controller-manager-76fdfbdbf4-w5r56" Jul 30 19:29:55.217988 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:29:55.217800 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0604020a-98cf-4d6c-8ee7-a12cbda5f73d-webhook-cert\") pod \"opendatahub-operator-controller-manager-76fdfbdbf4-w5r56\" (UID: \"0604020a-98cf-4d6c-8ee7-a12cbda5f73d\") " pod="opendatahub/opendatahub-operator-controller-manager-76fdfbdbf4-w5r56" Jul 30 19:29:55.217988 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:29:55.217828 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0604020a-98cf-4d6c-8ee7-a12cbda5f73d-manifests\") pod \"opendatahub-operator-controller-manager-76fdfbdbf4-w5r56\" (UID: \"0604020a-98cf-4d6c-8ee7-a12cbda5f73d\") " pod="opendatahub/opendatahub-operator-controller-manager-76fdfbdbf4-w5r56" Jul 30 19:29:55.217988 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:29:55.217952 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0604020a-98cf-4d6c-8ee7-a12cbda5f73d-tmp\") pod \"opendatahub-operator-controller-manager-76fdfbdbf4-w5r56\" (UID: \"0604020a-98cf-4d6c-8ee7-a12cbda5f73d\") " pod="opendatahub/opendatahub-operator-controller-manager-76fdfbdbf4-w5r56" Jul 30 19:29:55.217988 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:29:55.217983 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzfmq\" (UniqueName: \"kubernetes.io/projected/0604020a-98cf-4d6c-8ee7-a12cbda5f73d-kube-api-access-xzfmq\") pod \"opendatahub-operator-controller-manager-76fdfbdbf4-w5r56\" (UID: \"0604020a-98cf-4d6c-8ee7-a12cbda5f73d\") " pod="opendatahub/opendatahub-operator-controller-manager-76fdfbdbf4-w5r56" Jul 30 19:29:55.218247 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:29:55.218224 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0604020a-98cf-4d6c-8ee7-a12cbda5f73d-manifests\") pod \"opendatahub-operator-controller-manager-76fdfbdbf4-w5r56\" (UID: \"0604020a-98cf-4d6c-8ee7-a12cbda5f73d\") " pod="opendatahub/opendatahub-operator-controller-manager-76fdfbdbf4-w5r56" Jul 30 19:29:55.220219 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:29:55.220191 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0604020a-98cf-4d6c-8ee7-a12cbda5f73d-tmp\") pod \"opendatahub-operator-controller-manager-76fdfbdbf4-w5r56\" (UID: \"0604020a-98cf-4d6c-8ee7-a12cbda5f73d\") " pod="opendatahub/opendatahub-operator-controller-manager-76fdfbdbf4-w5r56" Jul 30 19:29:55.220343 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:29:55.220315 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0604020a-98cf-4d6c-8ee7-a12cbda5f73d-webhook-cert\") pod \"opendatahub-operator-controller-manager-76fdfbdbf4-w5r56\" (UID: \"0604020a-98cf-4d6c-8ee7-a12cbda5f73d\") " pod="opendatahub/opendatahub-operator-controller-manager-76fdfbdbf4-w5r56" Jul 30 19:29:55.220454 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:29:55.220436 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0604020a-98cf-4d6c-8ee7-a12cbda5f73d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76fdfbdbf4-w5r56\" (UID: \"0604020a-98cf-4d6c-8ee7-a12cbda5f73d\") " pod="opendatahub/opendatahub-operator-controller-manager-76fdfbdbf4-w5r56" Jul 30 19:29:55.229885 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:29:55.229856 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzfmq\" (UniqueName: \"kubernetes.io/projected/0604020a-98cf-4d6c-8ee7-a12cbda5f73d-kube-api-access-xzfmq\") pod \"opendatahub-operator-controller-manager-76fdfbdbf4-w5r56\" (UID: \"0604020a-98cf-4d6c-8ee7-a12cbda5f73d\") " pod="opendatahub/opendatahub-operator-controller-manager-76fdfbdbf4-w5r56" Jul 30 19:29:55.386210 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:29:55.386179 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-76fdfbdbf4-w5r56" Jul 30 19:29:55.531542 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:29:55.531514 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-76fdfbdbf4-w5r56"] Jul 30 19:29:55.533842 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:29:55.533811 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0604020a_98cf_4d6c_8ee7_a12cbda5f73d.slice/crio-1815680b4969b005fc588e3d6008b938e4593a462d756059754024960c858c3e WatchSource:0}: Error finding container 1815680b4969b005fc588e3d6008b938e4593a462d756059754024960c858c3e: Status 404 returned error can't find the container with id 1815680b4969b005fc588e3d6008b938e4593a462d756059754024960c858c3e Jul 30 19:29:55.535680 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:29:55.535661 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 30 19:29:55.723499 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:29:55.723375 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76fdfbdbf4-w5r56" event={"ID":"0604020a-98cf-4d6c-8ee7-a12cbda5f73d","Type":"ContainerStarted","Data":"1815680b4969b005fc588e3d6008b938e4593a462d756059754024960c858c3e"} Jul 30 19:29:58.740195 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:29:58.740158 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76fdfbdbf4-w5r56" event={"ID":"0604020a-98cf-4d6c-8ee7-a12cbda5f73d","Type":"ContainerStarted","Data":"79bc5a6ae476a1ddeb6c17b7745190a3a754b687c5f5f9e1669507f4a28d2370"} Jul 30 19:29:59.746976 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:29:59.746945 2556 generic.go:356] "Generic (PLEG): container finished" podID="0604020a-98cf-4d6c-8ee7-a12cbda5f73d" containerID="79bc5a6ae476a1ddeb6c17b7745190a3a754b687c5f5f9e1669507f4a28d2370" exitCode=0 Jul 30 19:29:59.747465 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:29:59.747028 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76fdfbdbf4-w5r56" event={"ID":"0604020a-98cf-4d6c-8ee7-a12cbda5f73d","Type":"ContainerDied","Data":"79bc5a6ae476a1ddeb6c17b7745190a3a754b687c5f5f9e1669507f4a28d2370"} Jul 30 19:30:00.754478 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:00.754441 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76fdfbdbf4-w5r56" event={"ID":"0604020a-98cf-4d6c-8ee7-a12cbda5f73d","Type":"ContainerStarted","Data":"eb679dccbfffcd7cc72149c9852c57cb422d3ac875a2618f31ff16239bf382e6"} Jul 30 19:30:00.754899 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:00.754558 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-76fdfbdbf4-w5r56" Jul 30 19:30:00.778866 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:00.778808 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-76fdfbdbf4-w5r56" podStartSLOduration=1.162635092 podStartE2EDuration="5.778790141s" totalImagesPullingTime="3.605300472s" totalInitContainerRuntime="1.010854577s" isStatefulPod=false podCreationTimestamp="2026-07-30 19:29:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:30:00.776854152 +0000 UTC m=+322.996413077" watchObservedRunningTime="2026-07-30 19:30:00.778790141 +0000 UTC m=+322.998349053" Jul 30 19:30:11.760765 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:11.760733 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-76fdfbdbf4-w5r56" Jul 30 19:30:15.009011 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:15.008970 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-ntx4s"] Jul 30 19:30:15.022174 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:15.022140 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-ntx4s"] Jul 30 19:30:15.022453 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:15.022286 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-ntx4s" Jul 30 19:30:15.024722 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:15.024698 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 30 19:30:15.024864 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:15.024809 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-kpggw\"" Jul 30 19:30:15.194744 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:15.194706 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slkvg\" (UniqueName: \"kubernetes.io/projected/ab7a30d6-2b2d-48f9-87d2-b773a81e2dfe-kube-api-access-slkvg\") pod \"odh-model-controller-7998c67dc4-ntx4s\" (UID: \"ab7a30d6-2b2d-48f9-87d2-b773a81e2dfe\") " pod="opendatahub/odh-model-controller-7998c67dc4-ntx4s" Jul 30 19:30:15.194941 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:15.194756 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab7a30d6-2b2d-48f9-87d2-b773a81e2dfe-cert\") pod \"odh-model-controller-7998c67dc4-ntx4s\" (UID: \"ab7a30d6-2b2d-48f9-87d2-b773a81e2dfe\") " pod="opendatahub/odh-model-controller-7998c67dc4-ntx4s" Jul 30 19:30:15.296148 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:15.296054 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slkvg\" (UniqueName: \"kubernetes.io/projected/ab7a30d6-2b2d-48f9-87d2-b773a81e2dfe-kube-api-access-slkvg\") pod \"odh-model-controller-7998c67dc4-ntx4s\" (UID: \"ab7a30d6-2b2d-48f9-87d2-b773a81e2dfe\") " pod="opendatahub/odh-model-controller-7998c67dc4-ntx4s" Jul 30 19:30:15.296148 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:15.296105 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab7a30d6-2b2d-48f9-87d2-b773a81e2dfe-cert\") pod \"odh-model-controller-7998c67dc4-ntx4s\" (UID: \"ab7a30d6-2b2d-48f9-87d2-b773a81e2dfe\") " pod="opendatahub/odh-model-controller-7998c67dc4-ntx4s" Jul 30 19:30:15.296343 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:30:15.296235 2556 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 30 19:30:15.296343 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:30:15.296295 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab7a30d6-2b2d-48f9-87d2-b773a81e2dfe-cert podName:ab7a30d6-2b2d-48f9-87d2-b773a81e2dfe nodeName:}" failed. No retries permitted until 2026-07-30 19:30:15.796276396 +0000 UTC m=+338.015835288 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/ab7a30d6-2b2d-48f9-87d2-b773a81e2dfe-cert") pod "odh-model-controller-7998c67dc4-ntx4s" (UID: "ab7a30d6-2b2d-48f9-87d2-b773a81e2dfe") : secret "odh-model-controller-webhook-cert" not found Jul 30 19:30:15.304726 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:15.304691 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slkvg\" (UniqueName: \"kubernetes.io/projected/ab7a30d6-2b2d-48f9-87d2-b773a81e2dfe-kube-api-access-slkvg\") pod \"odh-model-controller-7998c67dc4-ntx4s\" (UID: \"ab7a30d6-2b2d-48f9-87d2-b773a81e2dfe\") " pod="opendatahub/odh-model-controller-7998c67dc4-ntx4s" Jul 30 19:30:15.700520 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:15.700481 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-ldgsc"] Jul 30 19:30:15.703009 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:15.702992 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-ldgsc" Jul 30 19:30:15.706935 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:15.706910 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 30 19:30:15.724414 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:15.724345 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-ldgsc"] Jul 30 19:30:15.800494 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:15.800460 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8684d264-ac83-4de0-90a7-da2723d55c46-tls-certs\") pod \"model-serving-api-54d89bd5fc-ldgsc\" (UID: \"8684d264-ac83-4de0-90a7-da2723d55c46\") " pod="opendatahub/model-serving-api-54d89bd5fc-ldgsc" Jul 30 19:30:15.800709 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:15.800526 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff2fj\" (UniqueName: \"kubernetes.io/projected/8684d264-ac83-4de0-90a7-da2723d55c46-kube-api-access-ff2fj\") pod \"model-serving-api-54d89bd5fc-ldgsc\" (UID: \"8684d264-ac83-4de0-90a7-da2723d55c46\") " pod="opendatahub/model-serving-api-54d89bd5fc-ldgsc" Jul 30 19:30:15.800709 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:15.800626 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab7a30d6-2b2d-48f9-87d2-b773a81e2dfe-cert\") pod \"odh-model-controller-7998c67dc4-ntx4s\" (UID: \"ab7a30d6-2b2d-48f9-87d2-b773a81e2dfe\") " pod="opendatahub/odh-model-controller-7998c67dc4-ntx4s" Jul 30 19:30:15.802997 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:15.802968 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab7a30d6-2b2d-48f9-87d2-b773a81e2dfe-cert\") pod \"odh-model-controller-7998c67dc4-ntx4s\" (UID: \"ab7a30d6-2b2d-48f9-87d2-b773a81e2dfe\") " pod="opendatahub/odh-model-controller-7998c67dc4-ntx4s" Jul 30 19:30:15.902114 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:15.902073 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8684d264-ac83-4de0-90a7-da2723d55c46-tls-certs\") pod \"model-serving-api-54d89bd5fc-ldgsc\" (UID: \"8684d264-ac83-4de0-90a7-da2723d55c46\") " pod="opendatahub/model-serving-api-54d89bd5fc-ldgsc" Jul 30 19:30:15.902295 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:15.902123 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff2fj\" (UniqueName: \"kubernetes.io/projected/8684d264-ac83-4de0-90a7-da2723d55c46-kube-api-access-ff2fj\") pod \"model-serving-api-54d89bd5fc-ldgsc\" (UID: \"8684d264-ac83-4de0-90a7-da2723d55c46\") " pod="opendatahub/model-serving-api-54d89bd5fc-ldgsc" Jul 30 19:30:15.904596 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:15.904576 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8684d264-ac83-4de0-90a7-da2723d55c46-tls-certs\") pod \"model-serving-api-54d89bd5fc-ldgsc\" (UID: \"8684d264-ac83-4de0-90a7-da2723d55c46\") " pod="opendatahub/model-serving-api-54d89bd5fc-ldgsc" Jul 30 19:30:15.910779 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:15.910752 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff2fj\" (UniqueName: \"kubernetes.io/projected/8684d264-ac83-4de0-90a7-da2723d55c46-kube-api-access-ff2fj\") pod \"model-serving-api-54d89bd5fc-ldgsc\" (UID: \"8684d264-ac83-4de0-90a7-da2723d55c46\") " pod="opendatahub/model-serving-api-54d89bd5fc-ldgsc" Jul 30 19:30:15.934611 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:15.934575 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-ntx4s" Jul 30 19:30:16.014418 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:16.014123 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-ldgsc" Jul 30 19:30:16.087472 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:16.087438 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-ntx4s"] Jul 30 19:30:16.089698 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:30:16.089671 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab7a30d6_2b2d_48f9_87d2_b773a81e2dfe.slice/crio-3a3d24b8c8a6f07d0142049dfd5061038cba1a7a3857029548f872ccb79686dd WatchSource:0}: Error finding container 3a3d24b8c8a6f07d0142049dfd5061038cba1a7a3857029548f872ccb79686dd: Status 404 returned error can't find the container with id 3a3d24b8c8a6f07d0142049dfd5061038cba1a7a3857029548f872ccb79686dd Jul 30 19:30:16.159290 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:16.159077 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-ldgsc"] Jul 30 19:30:16.162259 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:30:16.162230 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8684d264_ac83_4de0_90a7_da2723d55c46.slice/crio-9a0f322ad6c4fcf24791228b0641cee3e077a5f9bb4bcc391044bc3ca3867929 WatchSource:0}: Error finding container 9a0f322ad6c4fcf24791228b0641cee3e077a5f9bb4bcc391044bc3ca3867929: Status 404 returned error can't find the container with id 9a0f322ad6c4fcf24791228b0641cee3e077a5f9bb4bcc391044bc3ca3867929 Jul 30 19:30:16.827547 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:16.827504 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-ntx4s" event={"ID":"ab7a30d6-2b2d-48f9-87d2-b773a81e2dfe","Type":"ContainerStarted","Data":"3a3d24b8c8a6f07d0142049dfd5061038cba1a7a3857029548f872ccb79686dd"} Jul 30 19:30:16.829119 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:16.829087 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-ldgsc" event={"ID":"8684d264-ac83-4de0-90a7-da2723d55c46","Type":"ContainerStarted","Data":"9a0f322ad6c4fcf24791228b0641cee3e077a5f9bb4bcc391044bc3ca3867929"} Jul 30 19:30:19.847566 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:19.847453 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-ldgsc" event={"ID":"8684d264-ac83-4de0-90a7-da2723d55c46","Type":"ContainerStarted","Data":"cfd62f0eeddfcf5eae921875633aebedef56541def926157e827b9ef769bc74f"} Jul 30 19:30:19.848009 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:19.847567 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-ldgsc" Jul 30 19:30:19.849006 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:19.848979 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-ntx4s" event={"ID":"ab7a30d6-2b2d-48f9-87d2-b773a81e2dfe","Type":"ContainerStarted","Data":"c6886e0f2cc41a264f3aea2f523d28d17a6230bc5731a0c62278a5ea73c3e3d9"} Jul 30 19:30:19.849104 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:19.849048 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-ntx4s" Jul 30 19:30:19.865626 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:19.865574 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-ldgsc" podStartSLOduration=1.627618217 podStartE2EDuration="4.865559082s" totalImagesPullingTime="3.237940865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 19:30:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:30:19.863922712 +0000 UTC m=+342.083481628" watchObservedRunningTime="2026-07-30 19:30:19.865559082 +0000 UTC m=+342.085117995" Jul 30 19:30:19.882478 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:19.882418 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-ntx4s" podStartSLOduration=2.567847491 podStartE2EDuration="5.882400897s" totalImagesPullingTime="3.314553406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 19:30:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:30:19.881038744 +0000 UTC m=+342.100597681" watchObservedRunningTime="2026-07-30 19:30:19.882400897 +0000 UTC m=+342.101959813" Jul 30 19:30:24.987562 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:24.987512 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-ccptj"] Jul 30 19:30:24.990284 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:24.990267 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ccptj" Jul 30 19:30:24.993363 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:24.993341 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-rfgjl\"" Jul 30 19:30:24.995142 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:24.995121 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 30 19:30:24.997562 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:24.997533 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86ec7077-e86c-4f03-ac13-8bded5e52536-cert\") pod \"kserve-controller-manager-d6f4b65b8-ccptj\" (UID: \"86ec7077-e86c-4f03-ac13-8bded5e52536\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ccptj" Jul 30 19:30:24.997697 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:24.997676 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smtbc\" (UniqueName: \"kubernetes.io/projected/86ec7077-e86c-4f03-ac13-8bded5e52536-kube-api-access-smtbc\") pod \"kserve-controller-manager-d6f4b65b8-ccptj\" (UID: \"86ec7077-e86c-4f03-ac13-8bded5e52536\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ccptj" Jul 30 19:30:25.013797 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:25.013761 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-ccptj"] Jul 30 19:30:25.040863 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:25.040814 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-vrmx5"] Jul 30 19:30:25.043823 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:25.043799 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vrmx5" Jul 30 19:30:25.047934 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:25.047917 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-fkw7g\"" Jul 30 19:30:25.048139 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:25.048126 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 30 19:30:25.068889 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:25.068856 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-vrmx5"] Jul 30 19:30:25.098171 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:25.098128 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwf67\" (UniqueName: \"kubernetes.io/projected/429bcbb5-b835-4e1a-9c76-f4512160f42e-kube-api-access-cwf67\") pod \"llmisvc-controller-manager-777c6bd7d5-vrmx5\" (UID: \"429bcbb5-b835-4e1a-9c76-f4512160f42e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vrmx5" Jul 30 19:30:25.098427 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:25.098178 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/429bcbb5-b835-4e1a-9c76-f4512160f42e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-vrmx5\" (UID: \"429bcbb5-b835-4e1a-9c76-f4512160f42e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vrmx5" Jul 30 19:30:25.098427 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:25.098271 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smtbc\" (UniqueName: \"kubernetes.io/projected/86ec7077-e86c-4f03-ac13-8bded5e52536-kube-api-access-smtbc\") pod \"kserve-controller-manager-d6f4b65b8-ccptj\" (UID: \"86ec7077-e86c-4f03-ac13-8bded5e52536\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ccptj" Jul 30 19:30:25.098427 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:25.098332 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86ec7077-e86c-4f03-ac13-8bded5e52536-cert\") pod \"kserve-controller-manager-d6f4b65b8-ccptj\" (UID: \"86ec7077-e86c-4f03-ac13-8bded5e52536\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ccptj" Jul 30 19:30:25.098607 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:30:25.098495 2556 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 30 19:30:25.098607 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:30:25.098558 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86ec7077-e86c-4f03-ac13-8bded5e52536-cert podName:86ec7077-e86c-4f03-ac13-8bded5e52536 nodeName:}" failed. No retries permitted until 2026-07-30 19:30:25.598539006 +0000 UTC m=+347.818097913 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/86ec7077-e86c-4f03-ac13-8bded5e52536-cert") pod "kserve-controller-manager-d6f4b65b8-ccptj" (UID: "86ec7077-e86c-4f03-ac13-8bded5e52536") : secret "kserve-webhook-server-cert" not found Jul 30 19:30:25.110496 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:25.110462 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smtbc\" (UniqueName: \"kubernetes.io/projected/86ec7077-e86c-4f03-ac13-8bded5e52536-kube-api-access-smtbc\") pod \"kserve-controller-manager-d6f4b65b8-ccptj\" (UID: \"86ec7077-e86c-4f03-ac13-8bded5e52536\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ccptj" Jul 30 19:30:25.198762 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:25.198723 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwf67\" (UniqueName: \"kubernetes.io/projected/429bcbb5-b835-4e1a-9c76-f4512160f42e-kube-api-access-cwf67\") pod \"llmisvc-controller-manager-777c6bd7d5-vrmx5\" (UID: \"429bcbb5-b835-4e1a-9c76-f4512160f42e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vrmx5" Jul 30 19:30:25.198762 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:25.198772 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/429bcbb5-b835-4e1a-9c76-f4512160f42e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-vrmx5\" (UID: \"429bcbb5-b835-4e1a-9c76-f4512160f42e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vrmx5" Jul 30 19:30:25.199036 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:30:25.198884 2556 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 30 19:30:25.199036 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:30:25.198939 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/429bcbb5-b835-4e1a-9c76-f4512160f42e-cert podName:429bcbb5-b835-4e1a-9c76-f4512160f42e nodeName:}" failed. No retries permitted until 2026-07-30 19:30:25.698923348 +0000 UTC m=+347.918482238 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/429bcbb5-b835-4e1a-9c76-f4512160f42e-cert") pod "llmisvc-controller-manager-777c6bd7d5-vrmx5" (UID: "429bcbb5-b835-4e1a-9c76-f4512160f42e") : secret "llmisvc-webhook-server-cert" not found Jul 30 19:30:25.212304 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:25.212269 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwf67\" (UniqueName: \"kubernetes.io/projected/429bcbb5-b835-4e1a-9c76-f4512160f42e-kube-api-access-cwf67\") pod \"llmisvc-controller-manager-777c6bd7d5-vrmx5\" (UID: \"429bcbb5-b835-4e1a-9c76-f4512160f42e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vrmx5" Jul 30 19:30:25.384054 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:25.384019 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5b4b6cc975-89d9s"] Jul 30 19:30:25.386589 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:25.386564 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5b4b6cc975-89d9s" Jul 30 19:30:25.389121 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:25.389098 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 30 19:30:25.389246 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:25.389167 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 30 19:30:25.389299 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:25.389239 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 30 19:30:25.389299 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:25.389241 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-6t64b\"" Jul 30 19:30:25.389528 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:25.389512 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 30 19:30:25.400282 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:25.400246 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7wp6\" (UniqueName: \"kubernetes.io/projected/b68a3551-b660-476a-84e5-66cb8328fa6b-kube-api-access-r7wp6\") pod \"kube-auth-proxy-5b4b6cc975-89d9s\" (UID: \"b68a3551-b660-476a-84e5-66cb8328fa6b\") " pod="openshift-ingress/kube-auth-proxy-5b4b6cc975-89d9s" Jul 30 19:30:25.400458 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:25.400434 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b68a3551-b660-476a-84e5-66cb8328fa6b-tmp\") pod \"kube-auth-proxy-5b4b6cc975-89d9s\" (UID: \"b68a3551-b660-476a-84e5-66cb8328fa6b\") " pod="openshift-ingress/kube-auth-proxy-5b4b6cc975-89d9s" Jul 30 19:30:25.400519 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:25.400492 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b68a3551-b660-476a-84e5-66cb8328fa6b-tls-certs\") pod \"kube-auth-proxy-5b4b6cc975-89d9s\" (UID: \"b68a3551-b660-476a-84e5-66cb8328fa6b\") " pod="openshift-ingress/kube-auth-proxy-5b4b6cc975-89d9s" Jul 30 19:30:25.401716 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:25.401688 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5b4b6cc975-89d9s"] Jul 30 19:30:25.501515 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:25.501464 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b68a3551-b660-476a-84e5-66cb8328fa6b-tmp\") pod \"kube-auth-proxy-5b4b6cc975-89d9s\" (UID: \"b68a3551-b660-476a-84e5-66cb8328fa6b\") " pod="openshift-ingress/kube-auth-proxy-5b4b6cc975-89d9s" Jul 30 19:30:25.501701 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:25.501536 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b68a3551-b660-476a-84e5-66cb8328fa6b-tls-certs\") pod \"kube-auth-proxy-5b4b6cc975-89d9s\" (UID: \"b68a3551-b660-476a-84e5-66cb8328fa6b\") " pod="openshift-ingress/kube-auth-proxy-5b4b6cc975-89d9s" Jul 30 19:30:25.501701 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:25.501589 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7wp6\" (UniqueName: \"kubernetes.io/projected/b68a3551-b660-476a-84e5-66cb8328fa6b-kube-api-access-r7wp6\") pod \"kube-auth-proxy-5b4b6cc975-89d9s\" (UID: \"b68a3551-b660-476a-84e5-66cb8328fa6b\") " pod="openshift-ingress/kube-auth-proxy-5b4b6cc975-89d9s" Jul 30 19:30:25.501701 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:30:25.501693 2556 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 30 19:30:25.501828 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:30:25.501759 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b68a3551-b660-476a-84e5-66cb8328fa6b-tls-certs podName:b68a3551-b660-476a-84e5-66cb8328fa6b nodeName:}" failed. No retries permitted until 2026-07-30 19:30:26.001743089 +0000 UTC m=+348.221301979 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b68a3551-b660-476a-84e5-66cb8328fa6b-tls-certs") pod "kube-auth-proxy-5b4b6cc975-89d9s" (UID: "b68a3551-b660-476a-84e5-66cb8328fa6b") : secret "kube-auth-proxy-tls" not found Jul 30 19:30:25.503739 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:25.503718 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b68a3551-b660-476a-84e5-66cb8328fa6b-tmp\") pod \"kube-auth-proxy-5b4b6cc975-89d9s\" (UID: \"b68a3551-b660-476a-84e5-66cb8328fa6b\") " pod="openshift-ingress/kube-auth-proxy-5b4b6cc975-89d9s" Jul 30 19:30:25.515773 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:25.515748 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7wp6\" (UniqueName: \"kubernetes.io/projected/b68a3551-b660-476a-84e5-66cb8328fa6b-kube-api-access-r7wp6\") pod \"kube-auth-proxy-5b4b6cc975-89d9s\" (UID: \"b68a3551-b660-476a-84e5-66cb8328fa6b\") " pod="openshift-ingress/kube-auth-proxy-5b4b6cc975-89d9s" Jul 30 19:30:25.602531 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:25.602491 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86ec7077-e86c-4f03-ac13-8bded5e52536-cert\") pod \"kserve-controller-manager-d6f4b65b8-ccptj\" (UID: \"86ec7077-e86c-4f03-ac13-8bded5e52536\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ccptj" Jul 30 19:30:25.604913 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:25.604892 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86ec7077-e86c-4f03-ac13-8bded5e52536-cert\") pod \"kserve-controller-manager-d6f4b65b8-ccptj\" (UID: \"86ec7077-e86c-4f03-ac13-8bded5e52536\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ccptj" Jul 30 19:30:25.704046 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:25.703954 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/429bcbb5-b835-4e1a-9c76-f4512160f42e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-vrmx5\" (UID: \"429bcbb5-b835-4e1a-9c76-f4512160f42e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vrmx5" Jul 30 19:30:25.706284 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:25.706265 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/429bcbb5-b835-4e1a-9c76-f4512160f42e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-vrmx5\" (UID: \"429bcbb5-b835-4e1a-9c76-f4512160f42e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vrmx5" Jul 30 19:30:25.901358 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:25.901318 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ccptj" Jul 30 19:30:25.954679 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:25.954604 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vrmx5" Jul 30 19:30:26.006542 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:26.006174 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b68a3551-b660-476a-84e5-66cb8328fa6b-tls-certs\") pod \"kube-auth-proxy-5b4b6cc975-89d9s\" (UID: \"b68a3551-b660-476a-84e5-66cb8328fa6b\") " pod="openshift-ingress/kube-auth-proxy-5b4b6cc975-89d9s" Jul 30 19:30:26.009915 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:26.009627 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b68a3551-b660-476a-84e5-66cb8328fa6b-tls-certs\") pod \"kube-auth-proxy-5b4b6cc975-89d9s\" (UID: \"b68a3551-b660-476a-84e5-66cb8328fa6b\") " pod="openshift-ingress/kube-auth-proxy-5b4b6cc975-89d9s" Jul 30 19:30:26.079673 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:26.078992 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-ccptj"] Jul 30 19:30:26.088642 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:30:26.088584 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86ec7077_e86c_4f03_ac13_8bded5e52536.slice/crio-976494f2f09c91644c089ba0d28a1a806ed7f99ba03aa63859e9dd5c1e533208 WatchSource:0}: Error finding container 976494f2f09c91644c089ba0d28a1a806ed7f99ba03aa63859e9dd5c1e533208: Status 404 returned error can't find the container with id 976494f2f09c91644c089ba0d28a1a806ed7f99ba03aa63859e9dd5c1e533208 Jul 30 19:30:26.164338 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:26.164311 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-vrmx5"] Jul 30 19:30:26.167299 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:30:26.167271 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod429bcbb5_b835_4e1a_9c76_f4512160f42e.slice/crio-52fa781c15dc031e680824e3ba589b5fe870cd10d1eb800de00da43637995e72 WatchSource:0}: Error finding container 52fa781c15dc031e680824e3ba589b5fe870cd10d1eb800de00da43637995e72: Status 404 returned error can't find the container with id 52fa781c15dc031e680824e3ba589b5fe870cd10d1eb800de00da43637995e72 Jul 30 19:30:26.296789 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:26.296699 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5b4b6cc975-89d9s" Jul 30 19:30:26.460833 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:26.460794 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5b4b6cc975-89d9s"] Jul 30 19:30:26.464310 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:30:26.464280 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb68a3551_b660_476a_84e5_66cb8328fa6b.slice/crio-ffad260e874d08c47375eec9fd0fe9872921657f21751323067339b3b7efcba9 WatchSource:0}: Error finding container ffad260e874d08c47375eec9fd0fe9872921657f21751323067339b3b7efcba9: Status 404 returned error can't find the container with id ffad260e874d08c47375eec9fd0fe9872921657f21751323067339b3b7efcba9 Jul 30 19:30:26.882866 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:26.882810 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ccptj" event={"ID":"86ec7077-e86c-4f03-ac13-8bded5e52536","Type":"ContainerStarted","Data":"976494f2f09c91644c089ba0d28a1a806ed7f99ba03aa63859e9dd5c1e533208"} Jul 30 19:30:26.884316 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:26.884280 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5b4b6cc975-89d9s" event={"ID":"b68a3551-b660-476a-84e5-66cb8328fa6b","Type":"ContainerStarted","Data":"ffad260e874d08c47375eec9fd0fe9872921657f21751323067339b3b7efcba9"} Jul 30 19:30:26.885513 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:26.885483 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vrmx5" event={"ID":"429bcbb5-b835-4e1a-9c76-f4512160f42e","Type":"ContainerStarted","Data":"52fa781c15dc031e680824e3ba589b5fe870cd10d1eb800de00da43637995e72"} Jul 30 19:30:27.265404 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:27.265311 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-6ftv2"] Jul 30 19:30:27.269899 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:27.269873 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6ftv2" Jul 30 19:30:27.273293 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:27.273270 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 30 19:30:27.273293 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:27.273275 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 30 19:30:27.273528 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:27.273318 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 30 19:30:27.273528 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:27.273270 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 30 19:30:27.273648 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:27.273588 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-qbhzj\"" Jul 30 19:30:27.291652 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:27.291628 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-6ftv2"] Jul 30 19:30:27.321182 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:27.321147 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fce6373e-9c39-45c0-9d46-a9c42a950783-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-6ftv2\" (UID: \"fce6373e-9c39-45c0-9d46-a9c42a950783\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6ftv2" Jul 30 19:30:27.321554 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:27.321205 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fce6373e-9c39-45c0-9d46-a9c42a950783-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-6ftv2\" (UID: \"fce6373e-9c39-45c0-9d46-a9c42a950783\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6ftv2" Jul 30 19:30:27.321554 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:27.321280 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lt74\" (UniqueName: \"kubernetes.io/projected/fce6373e-9c39-45c0-9d46-a9c42a950783-kube-api-access-8lt74\") pod \"istiod-openshift-gateway-5677ccb49b-6ftv2\" (UID: \"fce6373e-9c39-45c0-9d46-a9c42a950783\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6ftv2" Jul 30 19:30:27.321554 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:27.321430 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fce6373e-9c39-45c0-9d46-a9c42a950783-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-6ftv2\" (UID: \"fce6373e-9c39-45c0-9d46-a9c42a950783\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6ftv2" Jul 30 19:30:27.321554 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:27.321521 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fce6373e-9c39-45c0-9d46-a9c42a950783-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-6ftv2\" (UID: \"fce6373e-9c39-45c0-9d46-a9c42a950783\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6ftv2" Jul 30 19:30:27.321554 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:27.321546 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fce6373e-9c39-45c0-9d46-a9c42a950783-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-6ftv2\" (UID: \"fce6373e-9c39-45c0-9d46-a9c42a950783\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6ftv2" Jul 30 19:30:27.321851 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:27.321577 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fce6373e-9c39-45c0-9d46-a9c42a950783-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-6ftv2\" (UID: \"fce6373e-9c39-45c0-9d46-a9c42a950783\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6ftv2" Jul 30 19:30:27.422916 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:27.422883 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fce6373e-9c39-45c0-9d46-a9c42a950783-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-6ftv2\" (UID: \"fce6373e-9c39-45c0-9d46-a9c42a950783\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6ftv2" Jul 30 19:30:27.423099 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:27.422928 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fce6373e-9c39-45c0-9d46-a9c42a950783-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-6ftv2\" (UID: \"fce6373e-9c39-45c0-9d46-a9c42a950783\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6ftv2" Jul 30 19:30:27.423099 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:27.422958 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fce6373e-9c39-45c0-9d46-a9c42a950783-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-6ftv2\" (UID: \"fce6373e-9c39-45c0-9d46-a9c42a950783\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6ftv2" Jul 30 19:30:27.423099 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:27.423004 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fce6373e-9c39-45c0-9d46-a9c42a950783-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-6ftv2\" (UID: \"fce6373e-9c39-45c0-9d46-a9c42a950783\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6ftv2" Jul 30 19:30:27.423099 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:27.423037 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fce6373e-9c39-45c0-9d46-a9c42a950783-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-6ftv2\" (UID: \"fce6373e-9c39-45c0-9d46-a9c42a950783\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6ftv2" Jul 30 19:30:27.423099 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:27.423092 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lt74\" (UniqueName: \"kubernetes.io/projected/fce6373e-9c39-45c0-9d46-a9c42a950783-kube-api-access-8lt74\") pod \"istiod-openshift-gateway-5677ccb49b-6ftv2\" (UID: \"fce6373e-9c39-45c0-9d46-a9c42a950783\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6ftv2" Jul 30 19:30:27.423370 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:27.423171 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fce6373e-9c39-45c0-9d46-a9c42a950783-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-6ftv2\" (UID: \"fce6373e-9c39-45c0-9d46-a9c42a950783\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6ftv2" Jul 30 19:30:27.424372 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:27.424343 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fce6373e-9c39-45c0-9d46-a9c42a950783-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-6ftv2\" (UID: \"fce6373e-9c39-45c0-9d46-a9c42a950783\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6ftv2" Jul 30 19:30:27.429107 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:27.429080 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fce6373e-9c39-45c0-9d46-a9c42a950783-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-6ftv2\" (UID: \"fce6373e-9c39-45c0-9d46-a9c42a950783\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6ftv2" Jul 30 19:30:27.429660 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:27.429634 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fce6373e-9c39-45c0-9d46-a9c42a950783-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-6ftv2\" (UID: \"fce6373e-9c39-45c0-9d46-a9c42a950783\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6ftv2" Jul 30 19:30:27.430017 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:27.429995 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fce6373e-9c39-45c0-9d46-a9c42a950783-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-6ftv2\" (UID: \"fce6373e-9c39-45c0-9d46-a9c42a950783\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6ftv2" Jul 30 19:30:27.430591 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:27.430547 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fce6373e-9c39-45c0-9d46-a9c42a950783-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-6ftv2\" (UID: \"fce6373e-9c39-45c0-9d46-a9c42a950783\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6ftv2" Jul 30 19:30:27.435149 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:27.435120 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lt74\" (UniqueName: \"kubernetes.io/projected/fce6373e-9c39-45c0-9d46-a9c42a950783-kube-api-access-8lt74\") pod \"istiod-openshift-gateway-5677ccb49b-6ftv2\" (UID: \"fce6373e-9c39-45c0-9d46-a9c42a950783\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6ftv2" Jul 30 19:30:27.436133 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:27.436095 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fce6373e-9c39-45c0-9d46-a9c42a950783-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-6ftv2\" (UID: \"fce6373e-9c39-45c0-9d46-a9c42a950783\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6ftv2" Jul 30 19:30:27.585127 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:27.585033 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6ftv2" Jul 30 19:30:27.803236 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:27.803209 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-6ftv2"] Jul 30 19:30:27.806395 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:30:27.806353 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfce6373e_9c39_45c0_9d46_a9c42a950783.slice/crio-e955666ea2689ab55841fc093013d5d3cbaea7ccd53bc84cfec11d2afbb53280 WatchSource:0}: Error finding container e955666ea2689ab55841fc093013d5d3cbaea7ccd53bc84cfec11d2afbb53280: Status 404 returned error can't find the container with id e955666ea2689ab55841fc093013d5d3cbaea7ccd53bc84cfec11d2afbb53280 Jul 30 19:30:27.893175 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:27.893130 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6ftv2" event={"ID":"fce6373e-9c39-45c0-9d46-a9c42a950783","Type":"ContainerStarted","Data":"e955666ea2689ab55841fc093013d5d3cbaea7ccd53bc84cfec11d2afbb53280"} Jul 30 19:30:30.856062 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:30.856020 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-ntx4s" Jul 30 19:30:30.858700 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:30.858676 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-ldgsc" Jul 30 19:30:30.915356 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:30.915308 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ccptj" event={"ID":"86ec7077-e86c-4f03-ac13-8bded5e52536","Type":"ContainerStarted","Data":"5b2e4bf47a1c90deb10e544cf64f156108c8232c6d89e7535944b9112dc682ae"} Jul 30 19:30:30.915645 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:30.915622 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ccptj" Jul 30 19:30:30.918022 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:30.917992 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5b4b6cc975-89d9s" event={"ID":"b68a3551-b660-476a-84e5-66cb8328fa6b","Type":"ContainerStarted","Data":"4c109b53628196815cb0551639281b0ee225a7f165957f4f9bf163c292a2b92e"} Jul 30 19:30:30.920150 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:30.920120 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vrmx5" event={"ID":"429bcbb5-b835-4e1a-9c76-f4512160f42e","Type":"ContainerStarted","Data":"596d9e76b37868d3d329518d217c4f80d2f6deec149b08888cb065c40a819a91"} Jul 30 19:30:30.920746 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:30.920723 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vrmx5" Jul 30 19:30:30.933731 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:30.933684 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ccptj" podStartSLOduration=3.011612646 podStartE2EDuration="6.933668153s" totalImagesPullingTime="3.922055507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 19:30:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:30:30.931979841 +0000 UTC m=+353.151538755" watchObservedRunningTime="2026-07-30 19:30:30.933668153 +0000 UTC m=+353.153227066" Jul 30 19:30:30.948658 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:30.948614 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vrmx5" podStartSLOduration=2.094083451 podStartE2EDuration="5.948596025s" totalImagesPullingTime="3.854512574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 19:30:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:30:30.948304032 +0000 UTC m=+353.167862968" watchObservedRunningTime="2026-07-30 19:30:30.948596025 +0000 UTC m=+353.168154992" Jul 30 19:30:30.966522 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:30.966467 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5b4b6cc975-89d9s" podStartSLOduration=2.41269758 podStartE2EDuration="5.966445704s" totalImagesPullingTime="3.553748124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 19:30:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:30:30.964706155 +0000 UTC m=+353.184265070" watchObservedRunningTime="2026-07-30 19:30:30.966445704 +0000 UTC m=+353.186004618" Jul 30 19:30:31.943902 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:31.943856 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 30 19:30:31.944278 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:31.943946 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 30 19:30:32.935952 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:32.935912 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6ftv2" event={"ID":"fce6373e-9c39-45c0-9d46-a9c42a950783","Type":"ContainerStarted","Data":"95978a303783b90c3cd993ffc4b6dc663a21e1db2d4bbe756433fa36a6654627"} Jul 30 19:30:32.936125 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:32.935980 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6ftv2" Jul 30 19:30:32.961290 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:32.961239 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6ftv2" podStartSLOduration=1.82663545 podStartE2EDuration="5.9612227s" totalImagesPullingTime="4.13458725s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 19:30:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:30:32.959067947 +0000 UTC m=+355.178626865" watchObservedRunningTime="2026-07-30 19:30:32.9612227 +0000 UTC m=+355.180781613" Jul 30 19:30:33.942201 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:30:33.942172 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6ftv2" Jul 30 19:31:01.932269 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:31:01.932236 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ccptj" Jul 30 19:31:02.938618 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:31:02.938574 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vrmx5" Jul 30 19:31:59.368405 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:31:59.368346 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-vr5js"] Jul 30 19:31:59.370753 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:31:59.370737 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-vr5js" Jul 30 19:31:59.374328 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:31:59.374299 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-xmmk7\"" Jul 30 19:31:59.374466 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:31:59.374417 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 30 19:31:59.375249 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:31:59.375232 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 30 19:31:59.390631 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:31:59.390602 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-vr5js"] Jul 30 19:31:59.449064 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:31:59.449031 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnzmc\" (UniqueName: \"kubernetes.io/projected/5c5dee6c-0969-4dd4-8d79-0417dba7c5ad-kube-api-access-mnzmc\") pod \"authorino-operator-75cc8d5cb7-vr5js\" (UID: \"5c5dee6c-0969-4dd4-8d79-0417dba7c5ad\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-vr5js" Jul 30 19:31:59.549799 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:31:59.549766 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnzmc\" (UniqueName: \"kubernetes.io/projected/5c5dee6c-0969-4dd4-8d79-0417dba7c5ad-kube-api-access-mnzmc\") pod \"authorino-operator-75cc8d5cb7-vr5js\" (UID: \"5c5dee6c-0969-4dd4-8d79-0417dba7c5ad\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-vr5js" Jul 30 19:31:59.560396 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:31:59.560354 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnzmc\" (UniqueName: \"kubernetes.io/projected/5c5dee6c-0969-4dd4-8d79-0417dba7c5ad-kube-api-access-mnzmc\") pod \"authorino-operator-75cc8d5cb7-vr5js\" (UID: \"5c5dee6c-0969-4dd4-8d79-0417dba7c5ad\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-vr5js" Jul 30 19:31:59.681364 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:31:59.681258 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-vr5js" Jul 30 19:31:59.823474 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:31:59.823437 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-vr5js"] Jul 30 19:31:59.825845 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:31:59.825818 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c5dee6c_0969_4dd4_8d79_0417dba7c5ad.slice/crio-f1468dc47507ce95420b7fe8379ae18724fb127f95cb727af8ac65b534d2e0e3 WatchSource:0}: Error finding container f1468dc47507ce95420b7fe8379ae18724fb127f95cb727af8ac65b534d2e0e3: Status 404 returned error can't find the container with id f1468dc47507ce95420b7fe8379ae18724fb127f95cb727af8ac65b534d2e0e3 Jul 30 19:32:00.338872 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:32:00.338832 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-vr5js" event={"ID":"5c5dee6c-0969-4dd4-8d79-0417dba7c5ad","Type":"ContainerStarted","Data":"f1468dc47507ce95420b7fe8379ae18724fb127f95cb727af8ac65b534d2e0e3"} Jul 30 19:32:02.350728 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:32:02.350691 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-vr5js" event={"ID":"5c5dee6c-0969-4dd4-8d79-0417dba7c5ad","Type":"ContainerStarted","Data":"4785d45a19c0bc85654df0095e0f09b5e9a0b43da17eb6ba533577f877fb9773"} Jul 30 19:32:02.351145 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:32:02.350817 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-vr5js" Jul 30 19:32:02.383741 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:32:02.383686 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-vr5js" podStartSLOduration=0.999246387 podStartE2EDuration="3.383670917s" totalImagesPullingTime="2.38442453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 19:31:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:32:02.379815525 +0000 UTC m=+444.599374438" watchObservedRunningTime="2026-07-30 19:32:02.383670917 +0000 UTC m=+444.603229905" Jul 30 19:32:13.358152 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:32:13.358113 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-vr5js" Jul 30 19:32:14.516196 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:32:14.516162 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lbpw5"] Jul 30 19:32:14.519087 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:32:14.519062 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lbpw5" Jul 30 19:32:14.521815 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:32:14.521792 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-8lznw\"" Jul 30 19:32:14.532395 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:32:14.532364 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lbpw5"] Jul 30 19:32:14.584530 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:32:14.584492 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxphq\" (UniqueName: \"kubernetes.io/projected/90883584-bc52-4465-a02f-cf5f565636d0-kube-api-access-dxphq\") pod \"limitador-operator-controller-manager-859d7b78bd-lbpw5\" (UID: \"90883584-bc52-4465-a02f-cf5f565636d0\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lbpw5" Jul 30 19:32:14.685821 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:32:14.685788 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxphq\" (UniqueName: \"kubernetes.io/projected/90883584-bc52-4465-a02f-cf5f565636d0-kube-api-access-dxphq\") pod \"limitador-operator-controller-manager-859d7b78bd-lbpw5\" (UID: \"90883584-bc52-4465-a02f-cf5f565636d0\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lbpw5" Jul 30 19:32:14.696604 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:32:14.696572 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxphq\" (UniqueName: \"kubernetes.io/projected/90883584-bc52-4465-a02f-cf5f565636d0-kube-api-access-dxphq\") pod \"limitador-operator-controller-manager-859d7b78bd-lbpw5\" (UID: \"90883584-bc52-4465-a02f-cf5f565636d0\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lbpw5" Jul 30 19:32:14.832981 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:32:14.832874 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lbpw5" Jul 30 19:32:14.968293 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:32:14.968267 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lbpw5"] Jul 30 19:32:14.969946 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:32:14.969913 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90883584_bc52_4465_a02f_cf5f565636d0.slice/crio-b33db710a924386746ef7b7215bcd0bdd6fe424764a6fd4d468b228d428daef0 WatchSource:0}: Error finding container b33db710a924386746ef7b7215bcd0bdd6fe424764a6fd4d468b228d428daef0: Status 404 returned error can't find the container with id b33db710a924386746ef7b7215bcd0bdd6fe424764a6fd4d468b228d428daef0 Jul 30 19:32:15.412325 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:32:15.412288 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lbpw5" event={"ID":"90883584-bc52-4465-a02f-cf5f565636d0","Type":"ContainerStarted","Data":"b33db710a924386746ef7b7215bcd0bdd6fe424764a6fd4d468b228d428daef0"} Jul 30 19:32:17.426032 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:32:17.425996 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lbpw5" event={"ID":"90883584-bc52-4465-a02f-cf5f565636d0","Type":"ContainerStarted","Data":"5235c569663be1f3fc81b2ee91646c12136c03b506def1680f6f4b8492517a51"} Jul 30 19:32:17.426445 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:32:17.426139 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lbpw5" Jul 30 19:32:17.443995 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:32:17.443950 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lbpw5" podStartSLOduration=1.40987329 podStartE2EDuration="3.443932787s" totalImagesPullingTime="2.034059497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 19:32:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:32:17.44271515 +0000 UTC m=+459.662274064" watchObservedRunningTime="2026-07-30 19:32:17.443932787 +0000 UTC m=+459.663491700" Jul 30 19:32:28.432396 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:32:28.432350 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lbpw5" Jul 30 19:33:53.389818 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:33:53.389741 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-798f567bfd-nxz9k"] Jul 30 19:33:53.392256 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:33:53.392240 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-798f567bfd-nxz9k" Jul 30 19:33:53.394756 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:33:53.394738 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 30 19:33:53.394879 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:33:53.394856 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-6n8w5\"" Jul 30 19:33:53.394953 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:33:53.394883 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 30 19:33:53.403219 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:33:53.403192 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-798f567bfd-nxz9k"] Jul 30 19:33:53.465003 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:33:53.464967 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9c5865d6-59aa-43a9-a323-ed7a5f979e85-webhook-cert\") pod \"maas-controller-798f567bfd-nxz9k\" (UID: \"9c5865d6-59aa-43a9-a323-ed7a5f979e85\") " pod="opendatahub/maas-controller-798f567bfd-nxz9k" Jul 30 19:33:53.465178 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:33:53.465076 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc6g2\" (UniqueName: \"kubernetes.io/projected/9c5865d6-59aa-43a9-a323-ed7a5f979e85-kube-api-access-qc6g2\") pod \"maas-controller-798f567bfd-nxz9k\" (UID: \"9c5865d6-59aa-43a9-a323-ed7a5f979e85\") " pod="opendatahub/maas-controller-798f567bfd-nxz9k" Jul 30 19:33:53.566344 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:33:53.566310 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc6g2\" (UniqueName: \"kubernetes.io/projected/9c5865d6-59aa-43a9-a323-ed7a5f979e85-kube-api-access-qc6g2\") pod \"maas-controller-798f567bfd-nxz9k\" (UID: \"9c5865d6-59aa-43a9-a323-ed7a5f979e85\") " pod="opendatahub/maas-controller-798f567bfd-nxz9k" Jul 30 19:33:53.566574 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:33:53.566419 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9c5865d6-59aa-43a9-a323-ed7a5f979e85-webhook-cert\") pod \"maas-controller-798f567bfd-nxz9k\" (UID: \"9c5865d6-59aa-43a9-a323-ed7a5f979e85\") " pod="opendatahub/maas-controller-798f567bfd-nxz9k" Jul 30 19:33:53.569097 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:33:53.569070 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9c5865d6-59aa-43a9-a323-ed7a5f979e85-webhook-cert\") pod \"maas-controller-798f567bfd-nxz9k\" (UID: \"9c5865d6-59aa-43a9-a323-ed7a5f979e85\") " pod="opendatahub/maas-controller-798f567bfd-nxz9k" Jul 30 19:33:53.573715 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:33:53.573682 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc6g2\" (UniqueName: \"kubernetes.io/projected/9c5865d6-59aa-43a9-a323-ed7a5f979e85-kube-api-access-qc6g2\") pod \"maas-controller-798f567bfd-nxz9k\" (UID: \"9c5865d6-59aa-43a9-a323-ed7a5f979e85\") " pod="opendatahub/maas-controller-798f567bfd-nxz9k" Jul 30 19:33:53.703195 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:33:53.703099 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-798f567bfd-nxz9k" Jul 30 19:33:53.836757 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:33:53.836730 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-798f567bfd-nxz9k"] Jul 30 19:33:53.839141 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:33:53.839107 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c5865d6_59aa_43a9_a323_ed7a5f979e85.slice/crio-4bc4d45cc71cb4d7d496db8d9c78db2a75e7129ac53cffdf0560afb6020f2fb1 WatchSource:0}: Error finding container 4bc4d45cc71cb4d7d496db8d9c78db2a75e7129ac53cffdf0560afb6020f2fb1: Status 404 returned error can't find the container with id 4bc4d45cc71cb4d7d496db8d9c78db2a75e7129ac53cffdf0560afb6020f2fb1 Jul 30 19:33:53.887350 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:33:53.887311 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-798f567bfd-nxz9k" event={"ID":"9c5865d6-59aa-43a9-a323-ed7a5f979e85","Type":"ContainerStarted","Data":"4bc4d45cc71cb4d7d496db8d9c78db2a75e7129ac53cffdf0560afb6020f2fb1"} Jul 30 19:33:56.907202 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:33:56.907160 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-798f567bfd-nxz9k" event={"ID":"9c5865d6-59aa-43a9-a323-ed7a5f979e85","Type":"ContainerStarted","Data":"3265c948bc2d783b57f0d6459c9904cde4a2ed55a8f5fed4361d79ef39a51880"} Jul 30 19:33:56.907669 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:33:56.907274 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-798f567bfd-nxz9k" Jul 30 19:33:56.925257 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:33:56.925207 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-798f567bfd-nxz9k" podStartSLOduration=1.677466825 podStartE2EDuration="3.925191741s" totalImagesPullingTime="2.247724916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 19:33:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:33:56.923356332 +0000 UTC m=+559.142915246" watchObservedRunningTime="2026-07-30 19:33:56.925191741 +0000 UTC m=+559.144750653" Jul 30 19:34:07.528899 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:07.528863 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-798f567bfd-nxz9k"] Jul 30 19:34:07.529503 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:07.529127 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-798f567bfd-nxz9k" podUID="9c5865d6-59aa-43a9-a323-ed7a5f979e85" containerName="manager" containerID="cri-o://3265c948bc2d783b57f0d6459c9904cde4a2ed55a8f5fed4361d79ef39a51880" gracePeriod=30 Jul 30 19:34:07.531400 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:07.531362 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-798f567bfd-nxz9k" Jul 30 19:34:07.773556 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:07.773529 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-798f567bfd-nxz9k" Jul 30 19:34:07.902071 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:07.902033 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9c5865d6-59aa-43a9-a323-ed7a5f979e85-webhook-cert\") pod \"9c5865d6-59aa-43a9-a323-ed7a5f979e85\" (UID: \"9c5865d6-59aa-43a9-a323-ed7a5f979e85\") " Jul 30 19:34:07.902249 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:07.902161 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qc6g2\" (UniqueName: \"kubernetes.io/projected/9c5865d6-59aa-43a9-a323-ed7a5f979e85-kube-api-access-qc6g2\") pod \"9c5865d6-59aa-43a9-a323-ed7a5f979e85\" (UID: \"9c5865d6-59aa-43a9-a323-ed7a5f979e85\") " Jul 30 19:34:07.904324 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:07.904299 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c5865d6-59aa-43a9-a323-ed7a5f979e85-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "9c5865d6-59aa-43a9-a323-ed7a5f979e85" (UID: "9c5865d6-59aa-43a9-a323-ed7a5f979e85"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 19:34:07.904426 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:07.904345 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c5865d6-59aa-43a9-a323-ed7a5f979e85-kube-api-access-qc6g2" (OuterVolumeSpecName: "kube-api-access-qc6g2") pod "9c5865d6-59aa-43a9-a323-ed7a5f979e85" (UID: "9c5865d6-59aa-43a9-a323-ed7a5f979e85"). InnerVolumeSpecName "kube-api-access-qc6g2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 19:34:07.960439 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:07.960404 2556 generic.go:356] "Generic (PLEG): container finished" podID="9c5865d6-59aa-43a9-a323-ed7a5f979e85" containerID="3265c948bc2d783b57f0d6459c9904cde4a2ed55a8f5fed4361d79ef39a51880" exitCode=0 Jul 30 19:34:07.960439 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:07.960444 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-798f567bfd-nxz9k" Jul 30 19:34:07.960634 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:07.960487 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-798f567bfd-nxz9k" event={"ID":"9c5865d6-59aa-43a9-a323-ed7a5f979e85","Type":"ContainerDied","Data":"3265c948bc2d783b57f0d6459c9904cde4a2ed55a8f5fed4361d79ef39a51880"} Jul 30 19:34:07.960634 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:07.960577 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-798f567bfd-nxz9k" event={"ID":"9c5865d6-59aa-43a9-a323-ed7a5f979e85","Type":"ContainerDied","Data":"4bc4d45cc71cb4d7d496db8d9c78db2a75e7129ac53cffdf0560afb6020f2fb1"} Jul 30 19:34:07.960634 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:07.960605 2556 scope.go:117] "RemoveContainer" containerID="3265c948bc2d783b57f0d6459c9904cde4a2ed55a8f5fed4361d79ef39a51880" Jul 30 19:34:07.969971 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:07.969953 2556 scope.go:117] "RemoveContainer" containerID="3265c948bc2d783b57f0d6459c9904cde4a2ed55a8f5fed4361d79ef39a51880" Jul 30 19:34:07.970232 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:34:07.970209 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3265c948bc2d783b57f0d6459c9904cde4a2ed55a8f5fed4361d79ef39a51880\": container with ID starting with 3265c948bc2d783b57f0d6459c9904cde4a2ed55a8f5fed4361d79ef39a51880 not found: ID does not exist" containerID="3265c948bc2d783b57f0d6459c9904cde4a2ed55a8f5fed4361d79ef39a51880" Jul 30 19:34:07.970307 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:07.970240 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3265c948bc2d783b57f0d6459c9904cde4a2ed55a8f5fed4361d79ef39a51880"} err="failed to get container status \"3265c948bc2d783b57f0d6459c9904cde4a2ed55a8f5fed4361d79ef39a51880\": rpc error: code = NotFound desc = could not find container \"3265c948bc2d783b57f0d6459c9904cde4a2ed55a8f5fed4361d79ef39a51880\": container with ID starting with 3265c948bc2d783b57f0d6459c9904cde4a2ed55a8f5fed4361d79ef39a51880 not found: ID does not exist" Jul 30 19:34:07.981734 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:07.981708 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-798f567bfd-nxz9k"] Jul 30 19:34:07.986964 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:07.986936 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-798f567bfd-nxz9k"] Jul 30 19:34:08.003546 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:08.003519 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc6g2\" (UniqueName: \"kubernetes.io/projected/9c5865d6-59aa-43a9-a323-ed7a5f979e85-kube-api-access-qc6g2\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:34:08.003546 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:08.003544 2556 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9c5865d6-59aa-43a9-a323-ed7a5f979e85-webhook-cert\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:34:08.324876 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:08.324796 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c5865d6-59aa-43a9-a323-ed7a5f979e85" path="/var/lib/kubelet/pods/9c5865d6-59aa-43a9-a323-ed7a5f979e85/volumes" Jul 30 19:34:30.299974 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:30.299936 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-w2gbm"] Jul 30 19:34:30.300504 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:30.300400 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c5865d6-59aa-43a9-a323-ed7a5f979e85" containerName="manager" Jul 30 19:34:30.300504 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:30.300414 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c5865d6-59aa-43a9-a323-ed7a5f979e85" containerName="manager" Jul 30 19:34:30.300654 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:30.300504 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c5865d6-59aa-43a9-a323-ed7a5f979e85" containerName="manager" Jul 30 19:34:30.308053 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:30.308034 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-w2gbm" Jul 30 19:34:30.310751 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:30.310730 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 30 19:34:30.314294 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:30.314269 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-w2gbm"] Jul 30 19:34:30.327950 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:30.327925 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-knk59"] Jul 30 19:34:30.332236 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:30.332218 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-knk59" Jul 30 19:34:30.334767 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:30.334744 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-ht7wc\"" Jul 30 19:34:30.341007 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:30.340981 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-knk59"] Jul 30 19:34:30.413611 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:30.413573 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlp8s\" (UniqueName: \"kubernetes.io/projected/8a9ee543-86c7-4d8e-ad53-cef7ece4bbf1-kube-api-access-qlp8s\") pod \"payload-pre-processing-687dbb6cf6-w2gbm\" (UID: \"8a9ee543-86c7-4d8e-ad53-cef7ece4bbf1\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-w2gbm" Jul 30 19:34:30.413812 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:30.413793 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bea701bd-a892-44cf-bedf-cbb6bd64bb37-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-knk59\" (UID: \"bea701bd-a892-44cf-bedf-cbb6bd64bb37\") " pod="openshift-ingress/payload-processing-5cc6cd444b-knk59" Jul 30 19:34:30.413889 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:30.413871 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2ddt\" (UniqueName: \"kubernetes.io/projected/bea701bd-a892-44cf-bedf-cbb6bd64bb37-kube-api-access-h2ddt\") pod \"payload-processing-5cc6cd444b-knk59\" (UID: \"bea701bd-a892-44cf-bedf-cbb6bd64bb37\") " pod="openshift-ingress/payload-processing-5cc6cd444b-knk59" Jul 30 19:34:30.413950 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:30.413938 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a9ee543-86c7-4d8e-ad53-cef7ece4bbf1-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-w2gbm\" (UID: \"8a9ee543-86c7-4d8e-ad53-cef7ece4bbf1\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-w2gbm" Jul 30 19:34:30.514735 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:30.514698 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bea701bd-a892-44cf-bedf-cbb6bd64bb37-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-knk59\" (UID: \"bea701bd-a892-44cf-bedf-cbb6bd64bb37\") " pod="openshift-ingress/payload-processing-5cc6cd444b-knk59" Jul 30 19:34:30.514891 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:30.514752 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2ddt\" (UniqueName: \"kubernetes.io/projected/bea701bd-a892-44cf-bedf-cbb6bd64bb37-kube-api-access-h2ddt\") pod \"payload-processing-5cc6cd444b-knk59\" (UID: \"bea701bd-a892-44cf-bedf-cbb6bd64bb37\") " pod="openshift-ingress/payload-processing-5cc6cd444b-knk59" Jul 30 19:34:30.514891 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:30.514782 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a9ee543-86c7-4d8e-ad53-cef7ece4bbf1-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-w2gbm\" (UID: \"8a9ee543-86c7-4d8e-ad53-cef7ece4bbf1\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-w2gbm" Jul 30 19:34:30.514891 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:30.514832 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlp8s\" (UniqueName: \"kubernetes.io/projected/8a9ee543-86c7-4d8e-ad53-cef7ece4bbf1-kube-api-access-qlp8s\") pod \"payload-pre-processing-687dbb6cf6-w2gbm\" (UID: \"8a9ee543-86c7-4d8e-ad53-cef7ece4bbf1\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-w2gbm" Jul 30 19:34:30.515488 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:30.515468 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bea701bd-a892-44cf-bedf-cbb6bd64bb37-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-knk59\" (UID: \"bea701bd-a892-44cf-bedf-cbb6bd64bb37\") " pod="openshift-ingress/payload-processing-5cc6cd444b-knk59" Jul 30 19:34:30.515584 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:30.515566 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a9ee543-86c7-4d8e-ad53-cef7ece4bbf1-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-w2gbm\" (UID: \"8a9ee543-86c7-4d8e-ad53-cef7ece4bbf1\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-w2gbm" Jul 30 19:34:30.530620 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:30.530591 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlp8s\" (UniqueName: \"kubernetes.io/projected/8a9ee543-86c7-4d8e-ad53-cef7ece4bbf1-kube-api-access-qlp8s\") pod \"payload-pre-processing-687dbb6cf6-w2gbm\" (UID: \"8a9ee543-86c7-4d8e-ad53-cef7ece4bbf1\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-w2gbm" Jul 30 19:34:30.532569 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:30.532549 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2ddt\" (UniqueName: \"kubernetes.io/projected/bea701bd-a892-44cf-bedf-cbb6bd64bb37-kube-api-access-h2ddt\") pod \"payload-processing-5cc6cd444b-knk59\" (UID: \"bea701bd-a892-44cf-bedf-cbb6bd64bb37\") " pod="openshift-ingress/payload-processing-5cc6cd444b-knk59" Jul 30 19:34:30.620959 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:30.620924 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-w2gbm" Jul 30 19:34:30.646541 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:30.646513 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-knk59" Jul 30 19:34:30.789895 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:30.789871 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-knk59"] Jul 30 19:34:30.792099 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:34:30.792058 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbea701bd_a892_44cf_bedf_cbb6bd64bb37.slice/crio-7877d04a6aaa208e8ebd152a3b82105a5e0e38463ca9177527188f1f475f4fbc WatchSource:0}: Error finding container 7877d04a6aaa208e8ebd152a3b82105a5e0e38463ca9177527188f1f475f4fbc: Status 404 returned error can't find the container with id 7877d04a6aaa208e8ebd152a3b82105a5e0e38463ca9177527188f1f475f4fbc Jul 30 19:34:30.969306 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:30.969284 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-w2gbm"] Jul 30 19:34:30.971283 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:34:30.971253 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a9ee543_86c7_4d8e_ad53_cef7ece4bbf1.slice/crio-3bf9f666857eba7db169c634f212e0341db7c551b0a3027123212a31eb3f4506 WatchSource:0}: Error finding container 3bf9f666857eba7db169c634f212e0341db7c551b0a3027123212a31eb3f4506: Status 404 returned error can't find the container with id 3bf9f666857eba7db169c634f212e0341db7c551b0a3027123212a31eb3f4506 Jul 30 19:34:31.073497 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:31.073460 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-knk59" event={"ID":"bea701bd-a892-44cf-bedf-cbb6bd64bb37","Type":"ContainerStarted","Data":"7877d04a6aaa208e8ebd152a3b82105a5e0e38463ca9177527188f1f475f4fbc"} Jul 30 19:34:31.074613 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:31.074573 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-w2gbm" event={"ID":"8a9ee543-86c7-4d8e-ad53-cef7ece4bbf1","Type":"ContainerStarted","Data":"3bf9f666857eba7db169c634f212e0341db7c551b0a3027123212a31eb3f4506"} Jul 30 19:34:34.096692 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:34.096653 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-knk59" event={"ID":"bea701bd-a892-44cf-bedf-cbb6bd64bb37","Type":"ContainerStarted","Data":"9844262ba1e5f0c637dc817531d13419a82937f6e70254ebaddff330095c4e06"} Jul 30 19:34:34.097200 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:34.096783 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-knk59" Jul 30 19:34:34.098529 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:34.098343 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-w2gbm" event={"ID":"8a9ee543-86c7-4d8e-ad53-cef7ece4bbf1","Type":"ContainerStarted","Data":"a4f37b334389ba43eb142ded7d3daef9b4fc04339d96101af269b462b3e9721d"} Jul 30 19:34:34.098529 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:34.098420 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-w2gbm" Jul 30 19:34:34.112875 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:34.112833 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-knk59" podStartSLOduration=1.562270443 podStartE2EDuration="4.112818763s" totalImagesPullingTime="2.55054832s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 19:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:34:34.111222151 +0000 UTC m=+596.330781065" watchObservedRunningTime="2026-07-30 19:34:34.112818763 +0000 UTC m=+596.332377725" Jul 30 19:34:34.127181 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:34.127128 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-w2gbm" podStartSLOduration=1.745865156 podStartE2EDuration="4.1271122s" totalImagesPullingTime="2.381247044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 19:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:34:34.125425085 +0000 UTC m=+596.344983999" watchObservedRunningTime="2026-07-30 19:34:34.1271122 +0000 UTC m=+596.346671113" Jul 30 19:34:38.208366 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:38.208335 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-795qw_4cdd383f-6493-4d83-80fa-f4ce17e8416e/ovn-acl-logging/0.log" Jul 30 19:34:38.208860 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:38.208341 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-795qw_4cdd383f-6493-4d83-80fa-f4ce17e8416e/ovn-acl-logging/0.log" Jul 30 19:34:45.105472 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:45.105442 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-w2gbm" Jul 30 19:34:45.105931 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:34:45.105510 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-knk59" Jul 30 19:35:06.362149 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:06.362044 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-g9xdg"] Jul 30 19:35:06.370984 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:06.370962 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g9xdg" Jul 30 19:35:06.374078 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:06.374052 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-78nfb\"" Jul 30 19:35:06.374340 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:06.374142 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 30 19:35:06.374340 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:06.374142 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 30 19:35:06.374863 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:06.374841 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 30 19:35:06.375565 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:06.375545 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-g9xdg"] Jul 30 19:35:06.551790 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:06.551723 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/855f848a-64ad-4bae-9216-132d39196bbf-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g9xdg\" (UID: \"855f848a-64ad-4bae-9216-132d39196bbf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g9xdg" Jul 30 19:35:06.551982 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:06.551904 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/855f848a-64ad-4bae-9216-132d39196bbf-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g9xdg\" (UID: \"855f848a-64ad-4bae-9216-132d39196bbf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g9xdg" Jul 30 19:35:06.551982 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:06.551961 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftmqv\" (UniqueName: \"kubernetes.io/projected/855f848a-64ad-4bae-9216-132d39196bbf-kube-api-access-ftmqv\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g9xdg\" (UID: \"855f848a-64ad-4bae-9216-132d39196bbf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g9xdg" Jul 30 19:35:06.552107 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:06.552050 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/855f848a-64ad-4bae-9216-132d39196bbf-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g9xdg\" (UID: \"855f848a-64ad-4bae-9216-132d39196bbf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g9xdg" Jul 30 19:35:06.552107 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:06.552084 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/855f848a-64ad-4bae-9216-132d39196bbf-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g9xdg\" (UID: \"855f848a-64ad-4bae-9216-132d39196bbf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g9xdg" Jul 30 19:35:06.552190 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:06.552133 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/855f848a-64ad-4bae-9216-132d39196bbf-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g9xdg\" (UID: \"855f848a-64ad-4bae-9216-132d39196bbf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g9xdg" Jul 30 19:35:06.652920 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:06.652890 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftmqv\" (UniqueName: \"kubernetes.io/projected/855f848a-64ad-4bae-9216-132d39196bbf-kube-api-access-ftmqv\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g9xdg\" (UID: \"855f848a-64ad-4bae-9216-132d39196bbf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g9xdg" Jul 30 19:35:06.653086 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:06.652933 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/855f848a-64ad-4bae-9216-132d39196bbf-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g9xdg\" (UID: \"855f848a-64ad-4bae-9216-132d39196bbf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g9xdg" Jul 30 19:35:06.653086 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:06.652959 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/855f848a-64ad-4bae-9216-132d39196bbf-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g9xdg\" (UID: \"855f848a-64ad-4bae-9216-132d39196bbf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g9xdg" Jul 30 19:35:06.653086 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:06.652996 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/855f848a-64ad-4bae-9216-132d39196bbf-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g9xdg\" (UID: \"855f848a-64ad-4bae-9216-132d39196bbf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g9xdg" Jul 30 19:35:06.653086 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:06.653042 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/855f848a-64ad-4bae-9216-132d39196bbf-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g9xdg\" (UID: \"855f848a-64ad-4bae-9216-132d39196bbf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g9xdg" Jul 30 19:35:06.653290 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:06.653098 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/855f848a-64ad-4bae-9216-132d39196bbf-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g9xdg\" (UID: \"855f848a-64ad-4bae-9216-132d39196bbf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g9xdg" Jul 30 19:35:06.653400 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:06.653361 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/855f848a-64ad-4bae-9216-132d39196bbf-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g9xdg\" (UID: \"855f848a-64ad-4bae-9216-132d39196bbf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g9xdg" Jul 30 19:35:06.653446 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:06.653369 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/855f848a-64ad-4bae-9216-132d39196bbf-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g9xdg\" (UID: \"855f848a-64ad-4bae-9216-132d39196bbf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g9xdg" Jul 30 19:35:06.653491 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:06.653477 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/855f848a-64ad-4bae-9216-132d39196bbf-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g9xdg\" (UID: \"855f848a-64ad-4bae-9216-132d39196bbf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g9xdg" Jul 30 19:35:06.655252 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:06.655228 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/855f848a-64ad-4bae-9216-132d39196bbf-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g9xdg\" (UID: \"855f848a-64ad-4bae-9216-132d39196bbf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g9xdg" Jul 30 19:35:06.655449 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:06.655433 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/855f848a-64ad-4bae-9216-132d39196bbf-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g9xdg\" (UID: \"855f848a-64ad-4bae-9216-132d39196bbf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g9xdg" Jul 30 19:35:06.660216 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:06.660194 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftmqv\" (UniqueName: \"kubernetes.io/projected/855f848a-64ad-4bae-9216-132d39196bbf-kube-api-access-ftmqv\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g9xdg\" (UID: \"855f848a-64ad-4bae-9216-132d39196bbf\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g9xdg" Jul 30 19:35:06.683718 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:06.683694 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g9xdg" Jul 30 19:35:06.818932 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:06.818901 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-g9xdg"] Jul 30 19:35:06.820936 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:35:06.820909 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod855f848a_64ad_4bae_9216_132d39196bbf.slice/crio-918e3875ad8b5fb6ae01e584e824060449abbaba215188e894f567f7b76ecc7f WatchSource:0}: Error finding container 918e3875ad8b5fb6ae01e584e824060449abbaba215188e894f567f7b76ecc7f: Status 404 returned error can't find the container with id 918e3875ad8b5fb6ae01e584e824060449abbaba215188e894f567f7b76ecc7f Jul 30 19:35:06.822355 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:06.822337 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 30 19:35:07.270604 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:07.270566 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g9xdg" event={"ID":"855f848a-64ad-4bae-9216-132d39196bbf","Type":"ContainerStarted","Data":"918e3875ad8b5fb6ae01e584e824060449abbaba215188e894f567f7b76ecc7f"} Jul 30 19:35:08.277164 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:08.277081 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g9xdg" event={"ID":"855f848a-64ad-4bae-9216-132d39196bbf","Type":"ContainerStarted","Data":"24419bf422580825d64514e7867592c199d4a97c0ed2e70090d24fe289280775"} Jul 30 19:35:08.297177 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:08.297113 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g9xdg" podStartSLOduration=1.194052069 podStartE2EDuration="2.297097515s" totalImagesPullingTime="1.103045446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 19:35:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:35:08.295118741 +0000 UTC m=+630.514677655" watchObservedRunningTime="2026-07-30 19:35:08.297097515 +0000 UTC m=+630.516656436" Jul 30 19:35:16.684095 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:16.684051 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g9xdg" Jul 30 19:35:16.684534 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:16.684111 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g9xdg" Jul 30 19:35:16.696771 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:16.696747 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g9xdg" Jul 30 19:35:17.335654 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:17.335623 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g9xdg" Jul 30 19:35:25.572637 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:25.572599 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdqrz9"] Jul 30 19:35:25.575761 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:25.575743 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdqrz9" Jul 30 19:35:25.578100 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:25.578073 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 30 19:35:25.583402 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:25.583363 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdqrz9"] Jul 30 19:35:25.629812 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:25.629770 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e28b750-b501-4578-b0b6-c9aef9c76eec-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdqrz9\" (UID: \"6e28b750-b501-4578-b0b6-c9aef9c76eec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdqrz9" Jul 30 19:35:25.629979 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:25.629821 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e28b750-b501-4578-b0b6-c9aef9c76eec-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdqrz9\" (UID: \"6e28b750-b501-4578-b0b6-c9aef9c76eec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdqrz9" Jul 30 19:35:25.629979 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:25.629846 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e28b750-b501-4578-b0b6-c9aef9c76eec-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdqrz9\" (UID: \"6e28b750-b501-4578-b0b6-c9aef9c76eec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdqrz9" Jul 30 19:35:25.629979 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:25.629891 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e28b750-b501-4578-b0b6-c9aef9c76eec-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdqrz9\" (UID: \"6e28b750-b501-4578-b0b6-c9aef9c76eec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdqrz9" Jul 30 19:35:25.629979 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:25.629952 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvs6l\" (UniqueName: \"kubernetes.io/projected/6e28b750-b501-4578-b0b6-c9aef9c76eec-kube-api-access-xvs6l\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdqrz9\" (UID: \"6e28b750-b501-4578-b0b6-c9aef9c76eec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdqrz9" Jul 30 19:35:25.630114 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:25.630060 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e28b750-b501-4578-b0b6-c9aef9c76eec-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdqrz9\" (UID: \"6e28b750-b501-4578-b0b6-c9aef9c76eec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdqrz9" Jul 30 19:35:25.730876 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:25.730836 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e28b750-b501-4578-b0b6-c9aef9c76eec-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdqrz9\" (UID: \"6e28b750-b501-4578-b0b6-c9aef9c76eec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdqrz9" Jul 30 19:35:25.730876 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:25.730881 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e28b750-b501-4578-b0b6-c9aef9c76eec-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdqrz9\" (UID: \"6e28b750-b501-4578-b0b6-c9aef9c76eec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdqrz9" Jul 30 19:35:25.731107 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:25.731064 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e28b750-b501-4578-b0b6-c9aef9c76eec-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdqrz9\" (UID: \"6e28b750-b501-4578-b0b6-c9aef9c76eec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdqrz9" Jul 30 19:35:25.731164 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:25.731105 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e28b750-b501-4578-b0b6-c9aef9c76eec-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdqrz9\" (UID: \"6e28b750-b501-4578-b0b6-c9aef9c76eec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdqrz9" Jul 30 19:35:25.731224 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:25.731167 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e28b750-b501-4578-b0b6-c9aef9c76eec-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdqrz9\" (UID: \"6e28b750-b501-4578-b0b6-c9aef9c76eec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdqrz9" Jul 30 19:35:25.731224 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:25.731211 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvs6l\" (UniqueName: \"kubernetes.io/projected/6e28b750-b501-4578-b0b6-c9aef9c76eec-kube-api-access-xvs6l\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdqrz9\" (UID: \"6e28b750-b501-4578-b0b6-c9aef9c76eec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdqrz9" Jul 30 19:35:25.731449 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:25.731426 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e28b750-b501-4578-b0b6-c9aef9c76eec-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdqrz9\" (UID: \"6e28b750-b501-4578-b0b6-c9aef9c76eec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdqrz9" Jul 30 19:35:25.731563 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:25.731451 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e28b750-b501-4578-b0b6-c9aef9c76eec-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdqrz9\" (UID: \"6e28b750-b501-4578-b0b6-c9aef9c76eec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdqrz9" Jul 30 19:35:25.731563 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:25.731547 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e28b750-b501-4578-b0b6-c9aef9c76eec-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdqrz9\" (UID: \"6e28b750-b501-4578-b0b6-c9aef9c76eec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdqrz9" Jul 30 19:35:25.733243 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:25.733218 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e28b750-b501-4578-b0b6-c9aef9c76eec-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdqrz9\" (UID: \"6e28b750-b501-4578-b0b6-c9aef9c76eec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdqrz9" Jul 30 19:35:25.733441 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:25.733423 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e28b750-b501-4578-b0b6-c9aef9c76eec-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdqrz9\" (UID: \"6e28b750-b501-4578-b0b6-c9aef9c76eec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdqrz9" Jul 30 19:35:25.739003 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:25.738981 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvs6l\" (UniqueName: \"kubernetes.io/projected/6e28b750-b501-4578-b0b6-c9aef9c76eec-kube-api-access-xvs6l\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdqrz9\" (UID: \"6e28b750-b501-4578-b0b6-c9aef9c76eec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdqrz9" Jul 30 19:35:25.887191 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:25.887156 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdqrz9" Jul 30 19:35:26.019575 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:26.019548 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdqrz9"] Jul 30 19:35:26.021357 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:35:26.021332 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e28b750_b501_4578_b0b6_c9aef9c76eec.slice/crio-1799e52e56ea07bf6db873c85ac38c1d612a7feaa638c64210819ab0612e0395 WatchSource:0}: Error finding container 1799e52e56ea07bf6db873c85ac38c1d612a7feaa638c64210819ab0612e0395: Status 404 returned error can't find the container with id 1799e52e56ea07bf6db873c85ac38c1d612a7feaa638c64210819ab0612e0395 Jul 30 19:35:26.367800 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:26.367761 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdqrz9" event={"ID":"6e28b750-b501-4578-b0b6-c9aef9c76eec","Type":"ContainerStarted","Data":"10676fd098ddd5e23da2cf7021e94480fa51916ceba83dffb8cd8bc6cc55dd4b"} Jul 30 19:35:26.367963 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:26.367822 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdqrz9" event={"ID":"6e28b750-b501-4578-b0b6-c9aef9c76eec","Type":"ContainerStarted","Data":"1799e52e56ea07bf6db873c85ac38c1d612a7feaa638c64210819ab0612e0395"} Jul 30 19:35:26.385475 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:26.385378 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdqrz9" podStartSLOduration=1.224200775 podStartE2EDuration="1.385358882s" totalImagesPullingTime="161.158107ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 19:35:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:35:26.385140442 +0000 UTC m=+648.604699356" watchObservedRunningTime="2026-07-30 19:35:26.385358882 +0000 UTC m=+648.604917796" Jul 30 19:35:34.563587 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:34.563552 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-t7vh2"] Jul 30 19:35:34.567510 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:34.567483 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7vh2" Jul 30 19:35:34.570044 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:34.570019 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 30 19:35:34.576358 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:34.576330 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-t7vh2"] Jul 30 19:35:34.721760 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:34.721715 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b89d74c6-dbd3-4435-9834-689783959c45-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t7vh2\" (UID: \"b89d74c6-dbd3-4435-9834-689783959c45\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7vh2" Jul 30 19:35:34.721935 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:34.721787 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b89d74c6-dbd3-4435-9834-689783959c45-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t7vh2\" (UID: \"b89d74c6-dbd3-4435-9834-689783959c45\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7vh2" Jul 30 19:35:34.721935 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:34.721820 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b89d74c6-dbd3-4435-9834-689783959c45-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t7vh2\" (UID: \"b89d74c6-dbd3-4435-9834-689783959c45\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7vh2" Jul 30 19:35:34.721935 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:34.721845 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6wwv\" (UniqueName: \"kubernetes.io/projected/b89d74c6-dbd3-4435-9834-689783959c45-kube-api-access-p6wwv\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t7vh2\" (UID: \"b89d74c6-dbd3-4435-9834-689783959c45\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7vh2" Jul 30 19:35:34.722091 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:34.721975 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b89d74c6-dbd3-4435-9834-689783959c45-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t7vh2\" (UID: \"b89d74c6-dbd3-4435-9834-689783959c45\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7vh2" Jul 30 19:35:34.722091 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:34.722011 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b89d74c6-dbd3-4435-9834-689783959c45-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t7vh2\" (UID: \"b89d74c6-dbd3-4435-9834-689783959c45\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7vh2" Jul 30 19:35:34.822582 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:34.822496 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b89d74c6-dbd3-4435-9834-689783959c45-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t7vh2\" (UID: \"b89d74c6-dbd3-4435-9834-689783959c45\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7vh2" Jul 30 19:35:34.822582 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:34.822541 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b89d74c6-dbd3-4435-9834-689783959c45-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t7vh2\" (UID: \"b89d74c6-dbd3-4435-9834-689783959c45\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7vh2" Jul 30 19:35:34.822582 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:34.822571 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6wwv\" (UniqueName: \"kubernetes.io/projected/b89d74c6-dbd3-4435-9834-689783959c45-kube-api-access-p6wwv\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t7vh2\" (UID: \"b89d74c6-dbd3-4435-9834-689783959c45\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7vh2" Jul 30 19:35:34.822849 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:34.822630 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b89d74c6-dbd3-4435-9834-689783959c45-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t7vh2\" (UID: \"b89d74c6-dbd3-4435-9834-689783959c45\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7vh2" Jul 30 19:35:34.822849 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:34.822652 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b89d74c6-dbd3-4435-9834-689783959c45-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t7vh2\" (UID: \"b89d74c6-dbd3-4435-9834-689783959c45\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7vh2" Jul 30 19:35:34.822849 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:34.822672 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b89d74c6-dbd3-4435-9834-689783959c45-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t7vh2\" (UID: \"b89d74c6-dbd3-4435-9834-689783959c45\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7vh2" Jul 30 19:35:34.823149 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:34.823124 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b89d74c6-dbd3-4435-9834-689783959c45-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t7vh2\" (UID: \"b89d74c6-dbd3-4435-9834-689783959c45\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7vh2" Jul 30 19:35:34.823149 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:34.823138 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b89d74c6-dbd3-4435-9834-689783959c45-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t7vh2\" (UID: \"b89d74c6-dbd3-4435-9834-689783959c45\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7vh2" Jul 30 19:35:34.823264 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:34.823165 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b89d74c6-dbd3-4435-9834-689783959c45-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t7vh2\" (UID: \"b89d74c6-dbd3-4435-9834-689783959c45\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7vh2" Jul 30 19:35:34.824981 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:34.824960 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b89d74c6-dbd3-4435-9834-689783959c45-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t7vh2\" (UID: \"b89d74c6-dbd3-4435-9834-689783959c45\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7vh2" Jul 30 19:35:34.825159 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:34.825142 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b89d74c6-dbd3-4435-9834-689783959c45-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t7vh2\" (UID: \"b89d74c6-dbd3-4435-9834-689783959c45\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7vh2" Jul 30 19:35:34.830429 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:34.830407 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6wwv\" (UniqueName: \"kubernetes.io/projected/b89d74c6-dbd3-4435-9834-689783959c45-kube-api-access-p6wwv\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t7vh2\" (UID: \"b89d74c6-dbd3-4435-9834-689783959c45\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7vh2" Jul 30 19:35:34.879213 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:34.879172 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7vh2" Jul 30 19:35:35.019369 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:35.019342 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-t7vh2"] Jul 30 19:35:35.021411 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:35:35.021370 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb89d74c6_dbd3_4435_9834_689783959c45.slice/crio-a7d2b5919adc2b29beb98b87a28de7199698ba76859708c4473ad74f73374f8e WatchSource:0}: Error finding container a7d2b5919adc2b29beb98b87a28de7199698ba76859708c4473ad74f73374f8e: Status 404 returned error can't find the container with id a7d2b5919adc2b29beb98b87a28de7199698ba76859708c4473ad74f73374f8e Jul 30 19:35:35.414709 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:35.414671 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7vh2" event={"ID":"b89d74c6-dbd3-4435-9834-689783959c45","Type":"ContainerStarted","Data":"6c0a715a1b0293a3f1e2c02b44f6be2b1dbd76588aa2a7544e4924a082580bd5"} Jul 30 19:35:35.414887 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:35.414734 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7vh2" event={"ID":"b89d74c6-dbd3-4435-9834-689783959c45","Type":"ContainerStarted","Data":"a7d2b5919adc2b29beb98b87a28de7199698ba76859708c4473ad74f73374f8e"} Jul 30 19:35:35.433682 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:35.433628 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7vh2" podStartSLOduration=1.223102258 podStartE2EDuration="1.433609583s" totalImagesPullingTime="210.507325ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 19:35:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:35:35.432319833 +0000 UTC m=+657.651878746" watchObservedRunningTime="2026-07-30 19:35:35.433609583 +0000 UTC m=+657.653168496" Jul 30 19:35:35.888067 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:35.888030 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdqrz9" Jul 30 19:35:35.888588 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:35.888108 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdqrz9" Jul 30 19:35:35.900846 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:35.900817 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdqrz9" Jul 30 19:35:36.430870 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:36.430834 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdqrz9" Jul 30 19:35:44.880312 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:44.880268 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7vh2" Jul 30 19:35:44.880880 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:44.880336 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7vh2" Jul 30 19:35:44.893084 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:44.893056 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7vh2" Jul 30 19:35:45.475297 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:35:45.475266 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t7vh2" Jul 30 19:39:38.242960 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:39:38.242881 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-795qw_4cdd383f-6493-4d83-80fa-f4ce17e8416e/ovn-acl-logging/0.log" Jul 30 19:39:38.244474 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:39:38.244454 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-795qw_4cdd383f-6493-4d83-80fa-f4ce17e8416e/ovn-acl-logging/0.log" Jul 30 19:44:38.275685 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:44:38.275641 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-795qw_4cdd383f-6493-4d83-80fa-f4ce17e8416e/ovn-acl-logging/0.log" Jul 30 19:44:38.277644 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:44:38.277623 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-795qw_4cdd383f-6493-4d83-80fa-f4ce17e8416e/ovn-acl-logging/0.log" Jul 30 19:49:38.309369 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:49:38.309338 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-795qw_4cdd383f-6493-4d83-80fa-f4ce17e8416e/ovn-acl-logging/0.log" Jul 30 19:49:38.311364 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:49:38.311333 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-795qw_4cdd383f-6493-4d83-80fa-f4ce17e8416e/ovn-acl-logging/0.log" Jul 30 19:54:38.342144 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:54:38.342113 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-795qw_4cdd383f-6493-4d83-80fa-f4ce17e8416e/ovn-acl-logging/0.log" Jul 30 19:54:38.344522 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:54:38.344498 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-795qw_4cdd383f-6493-4d83-80fa-f4ce17e8416e/ovn-acl-logging/0.log" Jul 30 19:56:13.294180 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:13.294095 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6d3e9f39-55485c95c6-vstpf"] Jul 30 19:56:13.297668 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:13.297651 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d3e9f39-55485c95c6-vstpf" Jul 30 19:56:13.300597 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:13.300563 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-6d3e9f39\"" Jul 30 19:56:13.300898 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:13.300878 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-6d3e9f39-dockercfg-mnjmj\"" Jul 30 19:56:13.311596 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:13.311563 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6d3e9f39-55485c95c6-vstpf"] Jul 30 19:56:13.328693 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:13.328656 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6d3e9f39-779c4fccf4-h4vst"] Jul 30 19:56:13.332267 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:13.332244 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6d3e9f39-779c4fccf4-h4vst" Jul 30 19:56:13.334796 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:13.334769 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7957a3ff-1410-4a15-af81-ba7a11cf7812-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6d3e9f39-55485c95c6-vstpf\" (UID: \"7957a3ff-1410-4a15-af81-ba7a11cf7812\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6d3e9f39-55485c95c6-vstpf" Jul 30 19:56:13.334912 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:13.334818 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trd5q\" (UniqueName: \"kubernetes.io/projected/7957a3ff-1410-4a15-af81-ba7a11cf7812-kube-api-access-trd5q\") pod \"payload-pre-processing-e2e-ait-6d3e9f39-55485c95c6-vstpf\" (UID: \"7957a3ff-1410-4a15-af81-ba7a11cf7812\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6d3e9f39-55485c95c6-vstpf" Jul 30 19:56:13.341239 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:13.341214 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6d3e9f39-779c4fccf4-h4vst"] Jul 30 19:56:13.436253 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:13.436218 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trd5q\" (UniqueName: \"kubernetes.io/projected/7957a3ff-1410-4a15-af81-ba7a11cf7812-kube-api-access-trd5q\") pod \"payload-pre-processing-e2e-ait-6d3e9f39-55485c95c6-vstpf\" (UID: \"7957a3ff-1410-4a15-af81-ba7a11cf7812\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6d3e9f39-55485c95c6-vstpf" Jul 30 19:56:13.436473 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:13.436300 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvr8k\" (UniqueName: \"kubernetes.io/projected/7b1d2d37-22a1-4415-8255-29be48bec2f3-kube-api-access-wvr8k\") pod \"payload-processing-e2e-ait-6d3e9f39-779c4fccf4-h4vst\" (UID: \"7b1d2d37-22a1-4415-8255-29be48bec2f3\") " pod="openshift-ingress/payload-processing-e2e-ait-6d3e9f39-779c4fccf4-h4vst" Jul 30 19:56:13.436473 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:13.436346 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7957a3ff-1410-4a15-af81-ba7a11cf7812-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6d3e9f39-55485c95c6-vstpf\" (UID: \"7957a3ff-1410-4a15-af81-ba7a11cf7812\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6d3e9f39-55485c95c6-vstpf" Jul 30 19:56:13.436611 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:13.436477 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b1d2d37-22a1-4415-8255-29be48bec2f3-plugins-config-volume\") pod \"payload-processing-e2e-ait-6d3e9f39-779c4fccf4-h4vst\" (UID: \"7b1d2d37-22a1-4415-8255-29be48bec2f3\") " pod="openshift-ingress/payload-processing-e2e-ait-6d3e9f39-779c4fccf4-h4vst" Jul 30 19:56:13.437077 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:13.437056 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7957a3ff-1410-4a15-af81-ba7a11cf7812-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6d3e9f39-55485c95c6-vstpf\" (UID: \"7957a3ff-1410-4a15-af81-ba7a11cf7812\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6d3e9f39-55485c95c6-vstpf" Jul 30 19:56:13.444666 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:13.444645 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trd5q\" (UniqueName: \"kubernetes.io/projected/7957a3ff-1410-4a15-af81-ba7a11cf7812-kube-api-access-trd5q\") pod \"payload-pre-processing-e2e-ait-6d3e9f39-55485c95c6-vstpf\" (UID: \"7957a3ff-1410-4a15-af81-ba7a11cf7812\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6d3e9f39-55485c95c6-vstpf" Jul 30 19:56:13.537618 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:13.537584 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b1d2d37-22a1-4415-8255-29be48bec2f3-plugins-config-volume\") pod \"payload-processing-e2e-ait-6d3e9f39-779c4fccf4-h4vst\" (UID: \"7b1d2d37-22a1-4415-8255-29be48bec2f3\") " pod="openshift-ingress/payload-processing-e2e-ait-6d3e9f39-779c4fccf4-h4vst" Jul 30 19:56:13.537778 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:13.537661 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvr8k\" (UniqueName: \"kubernetes.io/projected/7b1d2d37-22a1-4415-8255-29be48bec2f3-kube-api-access-wvr8k\") pod \"payload-processing-e2e-ait-6d3e9f39-779c4fccf4-h4vst\" (UID: \"7b1d2d37-22a1-4415-8255-29be48bec2f3\") " pod="openshift-ingress/payload-processing-e2e-ait-6d3e9f39-779c4fccf4-h4vst" Jul 30 19:56:13.538342 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:13.538323 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b1d2d37-22a1-4415-8255-29be48bec2f3-plugins-config-volume\") pod \"payload-processing-e2e-ait-6d3e9f39-779c4fccf4-h4vst\" (UID: \"7b1d2d37-22a1-4415-8255-29be48bec2f3\") " pod="openshift-ingress/payload-processing-e2e-ait-6d3e9f39-779c4fccf4-h4vst" Jul 30 19:56:13.544982 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:13.544926 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvr8k\" (UniqueName: \"kubernetes.io/projected/7b1d2d37-22a1-4415-8255-29be48bec2f3-kube-api-access-wvr8k\") pod \"payload-processing-e2e-ait-6d3e9f39-779c4fccf4-h4vst\" (UID: \"7b1d2d37-22a1-4415-8255-29be48bec2f3\") " pod="openshift-ingress/payload-processing-e2e-ait-6d3e9f39-779c4fccf4-h4vst" Jul 30 19:56:13.608125 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:13.608084 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d3e9f39-55485c95c6-vstpf" Jul 30 19:56:13.644761 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:13.644729 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6d3e9f39-779c4fccf4-h4vst" Jul 30 19:56:13.794948 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:13.794917 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6d3e9f39-779c4fccf4-h4vst"] Jul 30 19:56:13.796457 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:56:13.796395 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b1d2d37_22a1_4415_8255_29be48bec2f3.slice/crio-d9dbc6ef23d2ec4618de32dc65995f16dbd67416a6869e184f9ee9bbcdc407e4 WatchSource:0}: Error finding container d9dbc6ef23d2ec4618de32dc65995f16dbd67416a6869e184f9ee9bbcdc407e4: Status 404 returned error can't find the container with id d9dbc6ef23d2ec4618de32dc65995f16dbd67416a6869e184f9ee9bbcdc407e4 Jul 30 19:56:13.958113 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:13.958091 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6d3e9f39-55485c95c6-vstpf"] Jul 30 19:56:13.960417 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:56:13.960356 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7957a3ff_1410_4a15_af81_ba7a11cf7812.slice/crio-26770a3fb5305a5855615121f52bdb05d96adc32c82df4b210f32fa3472a1da5 WatchSource:0}: Error finding container 26770a3fb5305a5855615121f52bdb05d96adc32c82df4b210f32fa3472a1da5: Status 404 returned error can't find the container with id 26770a3fb5305a5855615121f52bdb05d96adc32c82df4b210f32fa3472a1da5 Jul 30 19:56:14.568984 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:14.568947 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d3e9f39-55485c95c6-vstpf" event={"ID":"7957a3ff-1410-4a15-af81-ba7a11cf7812","Type":"ContainerStarted","Data":"98c8c50e125558966606f54d460d4a58f165c9ef0384a195766f65ba62206e4b"} Jul 30 19:56:14.569465 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:14.569129 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d3e9f39-55485c95c6-vstpf" Jul 30 19:56:14.569465 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:14.569162 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d3e9f39-55485c95c6-vstpf" event={"ID":"7957a3ff-1410-4a15-af81-ba7a11cf7812","Type":"ContainerStarted","Data":"26770a3fb5305a5855615121f52bdb05d96adc32c82df4b210f32fa3472a1da5"} Jul 30 19:56:14.570415 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:14.570391 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6d3e9f39-779c4fccf4-h4vst" event={"ID":"7b1d2d37-22a1-4415-8255-29be48bec2f3","Type":"ContainerStarted","Data":"84731bae4c99eed77492031d9da7949ee12f70ae2a1a3cbec87e2e73e06f8c29"} Jul 30 19:56:14.570500 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:14.570434 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6d3e9f39-779c4fccf4-h4vst" event={"ID":"7b1d2d37-22a1-4415-8255-29be48bec2f3","Type":"ContainerStarted","Data":"d9dbc6ef23d2ec4618de32dc65995f16dbd67416a6869e184f9ee9bbcdc407e4"} Jul 30 19:56:14.570500 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:14.570453 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-6d3e9f39-779c4fccf4-h4vst" Jul 30 19:56:14.584084 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:14.584040 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d3e9f39-55485c95c6-vstpf" podStartSLOduration=1.5840280789999999 podStartE2EDuration="1.584028079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 19:56:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:56:14.583196345 +0000 UTC m=+1896.802755259" watchObservedRunningTime="2026-07-30 19:56:14.584028079 +0000 UTC m=+1896.803586992" Jul 30 19:56:14.598909 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:14.598872 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-6d3e9f39-779c4fccf4-h4vst" podStartSLOduration=1.598860282 podStartE2EDuration="1.598860282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 19:56:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:56:14.597283501 +0000 UTC m=+1896.816842426" watchObservedRunningTime="2026-07-30 19:56:14.598860282 +0000 UTC m=+1896.818419194" Jul 30 19:56:25.580080 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:25.580048 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d3e9f39-55485c95c6-vstpf" Jul 30 19:56:25.580532 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:25.580305 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-6d3e9f39-779c4fccf4-h4vst" Jul 30 19:56:43.302264 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:43.302227 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6d3e9f39-779c4fccf4-h4vst"] Jul 30 19:56:43.304711 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:43.302507 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-6d3e9f39-779c4fccf4-h4vst" podUID="7b1d2d37-22a1-4415-8255-29be48bec2f3" containerName="payload-processing" containerID="cri-o://84731bae4c99eed77492031d9da7949ee12f70ae2a1a3cbec87e2e73e06f8c29" gracePeriod=30 Jul 30 19:56:43.314907 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:43.314490 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6d3e9f39-55485c95c6-vstpf"] Jul 30 19:56:43.314907 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:43.314739 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d3e9f39-55485c95c6-vstpf" podUID="7957a3ff-1410-4a15-af81-ba7a11cf7812" containerName="payload-pre-processing" containerID="cri-o://98c8c50e125558966606f54d460d4a58f165c9ef0384a195766f65ba62206e4b" gracePeriod=30 Jul 30 19:56:43.608837 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:43.608816 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6d3e9f39-779c4fccf4-h4vst" Jul 30 19:56:43.612536 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:43.612517 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d3e9f39-55485c95c6-vstpf" Jul 30 19:56:43.623393 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:43.623365 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b1d2d37-22a1-4415-8255-29be48bec2f3-plugins-config-volume\") pod \"7b1d2d37-22a1-4415-8255-29be48bec2f3\" (UID: \"7b1d2d37-22a1-4415-8255-29be48bec2f3\") " Jul 30 19:56:43.623529 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:43.623516 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7957a3ff-1410-4a15-af81-ba7a11cf7812-plugins-config-volume\") pod \"7957a3ff-1410-4a15-af81-ba7a11cf7812\" (UID: \"7957a3ff-1410-4a15-af81-ba7a11cf7812\") " Jul 30 19:56:43.623568 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:43.623554 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-trd5q\" (UniqueName: \"kubernetes.io/projected/7957a3ff-1410-4a15-af81-ba7a11cf7812-kube-api-access-trd5q\") pod \"7957a3ff-1410-4a15-af81-ba7a11cf7812\" (UID: \"7957a3ff-1410-4a15-af81-ba7a11cf7812\") " Jul 30 19:56:43.623614 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:43.623573 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvr8k\" (UniqueName: \"kubernetes.io/projected/7b1d2d37-22a1-4415-8255-29be48bec2f3-kube-api-access-wvr8k\") pod \"7b1d2d37-22a1-4415-8255-29be48bec2f3\" (UID: \"7b1d2d37-22a1-4415-8255-29be48bec2f3\") " Jul 30 19:56:43.623739 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:43.623712 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b1d2d37-22a1-4415-8255-29be48bec2f3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7b1d2d37-22a1-4415-8255-29be48bec2f3" (UID: "7b1d2d37-22a1-4415-8255-29be48bec2f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 19:56:43.623865 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:43.623846 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7957a3ff-1410-4a15-af81-ba7a11cf7812-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7957a3ff-1410-4a15-af81-ba7a11cf7812" (UID: "7957a3ff-1410-4a15-af81-ba7a11cf7812"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 19:56:43.623929 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:43.623913 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b1d2d37-22a1-4415-8255-29be48bec2f3-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:56:43.626372 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:43.626349 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b1d2d37-22a1-4415-8255-29be48bec2f3-kube-api-access-wvr8k" (OuterVolumeSpecName: "kube-api-access-wvr8k") pod "7b1d2d37-22a1-4415-8255-29be48bec2f3" (UID: "7b1d2d37-22a1-4415-8255-29be48bec2f3"). InnerVolumeSpecName "kube-api-access-wvr8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 19:56:43.626490 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:43.626409 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7957a3ff-1410-4a15-af81-ba7a11cf7812-kube-api-access-trd5q" (OuterVolumeSpecName: "kube-api-access-trd5q") pod "7957a3ff-1410-4a15-af81-ba7a11cf7812" (UID: "7957a3ff-1410-4a15-af81-ba7a11cf7812"). InnerVolumeSpecName "kube-api-access-trd5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 19:56:43.716569 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:43.716533 2556 generic.go:356] "Generic (PLEG): container finished" podID="7957a3ff-1410-4a15-af81-ba7a11cf7812" containerID="98c8c50e125558966606f54d460d4a58f165c9ef0384a195766f65ba62206e4b" exitCode=0 Jul 30 19:56:43.716748 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:43.716576 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d3e9f39-55485c95c6-vstpf" Jul 30 19:56:43.716748 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:43.716622 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d3e9f39-55485c95c6-vstpf" event={"ID":"7957a3ff-1410-4a15-af81-ba7a11cf7812","Type":"ContainerDied","Data":"98c8c50e125558966606f54d460d4a58f165c9ef0384a195766f65ba62206e4b"} Jul 30 19:56:43.716748 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:43.716677 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6d3e9f39-55485c95c6-vstpf" event={"ID":"7957a3ff-1410-4a15-af81-ba7a11cf7812","Type":"ContainerDied","Data":"26770a3fb5305a5855615121f52bdb05d96adc32c82df4b210f32fa3472a1da5"} Jul 30 19:56:43.716748 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:43.716706 2556 scope.go:117] "RemoveContainer" containerID="98c8c50e125558966606f54d460d4a58f165c9ef0384a195766f65ba62206e4b" Jul 30 19:56:43.717966 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:43.717943 2556 generic.go:356] "Generic (PLEG): container finished" podID="7b1d2d37-22a1-4415-8255-29be48bec2f3" containerID="84731bae4c99eed77492031d9da7949ee12f70ae2a1a3cbec87e2e73e06f8c29" exitCode=0 Jul 30 19:56:43.718054 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:43.717989 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6d3e9f39-779c4fccf4-h4vst" Jul 30 19:56:43.718054 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:43.718014 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6d3e9f39-779c4fccf4-h4vst" event={"ID":"7b1d2d37-22a1-4415-8255-29be48bec2f3","Type":"ContainerDied","Data":"84731bae4c99eed77492031d9da7949ee12f70ae2a1a3cbec87e2e73e06f8c29"} Jul 30 19:56:43.718147 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:43.718055 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6d3e9f39-779c4fccf4-h4vst" event={"ID":"7b1d2d37-22a1-4415-8255-29be48bec2f3","Type":"ContainerDied","Data":"d9dbc6ef23d2ec4618de32dc65995f16dbd67416a6869e184f9ee9bbcdc407e4"} Jul 30 19:56:43.730655 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:43.726314 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7957a3ff-1410-4a15-af81-ba7a11cf7812-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:56:43.730655 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:43.726346 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trd5q\" (UniqueName: \"kubernetes.io/projected/7957a3ff-1410-4a15-af81-ba7a11cf7812-kube-api-access-trd5q\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:56:43.730655 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:43.726360 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvr8k\" (UniqueName: \"kubernetes.io/projected/7b1d2d37-22a1-4415-8255-29be48bec2f3-kube-api-access-wvr8k\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:56:43.730655 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:43.727071 2556 scope.go:117] "RemoveContainer" containerID="98c8c50e125558966606f54d460d4a58f165c9ef0384a195766f65ba62206e4b" Jul 30 19:56:43.730655 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:56:43.727686 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98c8c50e125558966606f54d460d4a58f165c9ef0384a195766f65ba62206e4b\": container with ID starting with 98c8c50e125558966606f54d460d4a58f165c9ef0384a195766f65ba62206e4b not found: ID does not exist" containerID="98c8c50e125558966606f54d460d4a58f165c9ef0384a195766f65ba62206e4b" Jul 30 19:56:43.730655 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:43.727727 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98c8c50e125558966606f54d460d4a58f165c9ef0384a195766f65ba62206e4b"} err="failed to get container status \"98c8c50e125558966606f54d460d4a58f165c9ef0384a195766f65ba62206e4b\": rpc error: code = NotFound desc = could not find container \"98c8c50e125558966606f54d460d4a58f165c9ef0384a195766f65ba62206e4b\": container with ID starting with 98c8c50e125558966606f54d460d4a58f165c9ef0384a195766f65ba62206e4b not found: ID does not exist" Jul 30 19:56:43.730655 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:43.727751 2556 scope.go:117] "RemoveContainer" containerID="84731bae4c99eed77492031d9da7949ee12f70ae2a1a3cbec87e2e73e06f8c29" Jul 30 19:56:43.739003 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:43.738987 2556 scope.go:117] "RemoveContainer" containerID="84731bae4c99eed77492031d9da7949ee12f70ae2a1a3cbec87e2e73e06f8c29" Jul 30 19:56:43.739236 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:56:43.739217 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84731bae4c99eed77492031d9da7949ee12f70ae2a1a3cbec87e2e73e06f8c29\": container with ID starting with 84731bae4c99eed77492031d9da7949ee12f70ae2a1a3cbec87e2e73e06f8c29 not found: ID does not exist" containerID="84731bae4c99eed77492031d9da7949ee12f70ae2a1a3cbec87e2e73e06f8c29" Jul 30 19:56:43.739286 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:43.739243 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84731bae4c99eed77492031d9da7949ee12f70ae2a1a3cbec87e2e73e06f8c29"} err="failed to get container status \"84731bae4c99eed77492031d9da7949ee12f70ae2a1a3cbec87e2e73e06f8c29\": rpc error: code = NotFound desc = could not find container \"84731bae4c99eed77492031d9da7949ee12f70ae2a1a3cbec87e2e73e06f8c29\": container with ID starting with 84731bae4c99eed77492031d9da7949ee12f70ae2a1a3cbec87e2e73e06f8c29 not found: ID does not exist" Jul 30 19:56:43.748454 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:43.748433 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6d3e9f39-55485c95c6-vstpf"] Jul 30 19:56:43.752567 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:43.752548 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6d3e9f39-55485c95c6-vstpf"] Jul 30 19:56:43.762154 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:43.762132 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6d3e9f39-779c4fccf4-h4vst"] Jul 30 19:56:43.765938 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:43.765920 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6d3e9f39-779c4fccf4-h4vst"] Jul 30 19:56:44.325689 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:44.325659 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7957a3ff-1410-4a15-af81-ba7a11cf7812" path="/var/lib/kubelet/pods/7957a3ff-1410-4a15-af81-ba7a11cf7812/volumes" Jul 30 19:56:44.326034 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:44.326021 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b1d2d37-22a1-4415-8255-29be48bec2f3" path="/var/lib/kubelet/pods/7b1d2d37-22a1-4415-8255-29be48bec2f3/volumes" Jul 30 19:56:50.058923 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:50.058888 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-57cec089-57b78ffbcb-lfhcx"] Jul 30 19:56:50.059357 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:50.059296 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7957a3ff-1410-4a15-af81-ba7a11cf7812" containerName="payload-pre-processing" Jul 30 19:56:50.059357 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:50.059307 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7957a3ff-1410-4a15-af81-ba7a11cf7812" containerName="payload-pre-processing" Jul 30 19:56:50.059357 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:50.059337 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b1d2d37-22a1-4415-8255-29be48bec2f3" containerName="payload-processing" Jul 30 19:56:50.059357 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:50.059343 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b1d2d37-22a1-4415-8255-29be48bec2f3" containerName="payload-processing" Jul 30 19:56:50.059574 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:50.059453 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b1d2d37-22a1-4415-8255-29be48bec2f3" containerName="payload-processing" Jul 30 19:56:50.059574 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:50.059463 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7957a3ff-1410-4a15-af81-ba7a11cf7812" containerName="payload-pre-processing" Jul 30 19:56:50.064150 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:50.064132 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-57cec089-57b78ffbcb-lfhcx" Jul 30 19:56:50.066370 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:50.066351 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-57cec089\"" Jul 30 19:56:50.072343 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:50.072291 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-57cec089-57b78ffbcb-lfhcx"] Jul 30 19:56:50.083651 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:50.083628 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/080673da-c4bc-4568-bcaa-07e0bea5ad87-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-57cec089-57b78ffbcb-lfhcx\" (UID: \"080673da-c4bc-4568-bcaa-07e0bea5ad87\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-57cec089-57b78ffbcb-lfhcx" Jul 30 19:56:50.083750 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:50.083711 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pspsj\" (UniqueName: \"kubernetes.io/projected/080673da-c4bc-4568-bcaa-07e0bea5ad87-kube-api-access-pspsj\") pod \"payload-pre-processing-e2e-migrate-57cec089-57b78ffbcb-lfhcx\" (UID: \"080673da-c4bc-4568-bcaa-07e0bea5ad87\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-57cec089-57b78ffbcb-lfhcx" Jul 30 19:56:50.094927 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:50.094902 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-57cec089-7dc4f9944f-428t8"] Jul 30 19:56:50.098805 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:50.098772 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-57cec089-7dc4f9944f-428t8" Jul 30 19:56:50.106136 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:50.106094 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-57cec089-dockercfg-snckf\"" Jul 30 19:56:50.109972 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:50.109911 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-57cec089-7dc4f9944f-428t8"] Jul 30 19:56:50.185273 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:50.185246 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pspsj\" (UniqueName: \"kubernetes.io/projected/080673da-c4bc-4568-bcaa-07e0bea5ad87-kube-api-access-pspsj\") pod \"payload-pre-processing-e2e-migrate-57cec089-57b78ffbcb-lfhcx\" (UID: \"080673da-c4bc-4568-bcaa-07e0bea5ad87\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-57cec089-57b78ffbcb-lfhcx" Jul 30 19:56:50.185452 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:50.185318 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/080673da-c4bc-4568-bcaa-07e0bea5ad87-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-57cec089-57b78ffbcb-lfhcx\" (UID: \"080673da-c4bc-4568-bcaa-07e0bea5ad87\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-57cec089-57b78ffbcb-lfhcx" Jul 30 19:56:50.185452 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:50.185349 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtlmx\" (UniqueName: \"kubernetes.io/projected/8b5fd0dc-5af2-41d1-9ca9-4b21bca3cdfb-kube-api-access-xtlmx\") pod \"payload-processing-e2e-migrate-57cec089-7dc4f9944f-428t8\" (UID: \"8b5fd0dc-5af2-41d1-9ca9-4b21bca3cdfb\") " pod="openshift-ingress/payload-processing-e2e-migrate-57cec089-7dc4f9944f-428t8" Jul 30 19:56:50.185452 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:50.185395 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b5fd0dc-5af2-41d1-9ca9-4b21bca3cdfb-plugins-config-volume\") pod \"payload-processing-e2e-migrate-57cec089-7dc4f9944f-428t8\" (UID: \"8b5fd0dc-5af2-41d1-9ca9-4b21bca3cdfb\") " pod="openshift-ingress/payload-processing-e2e-migrate-57cec089-7dc4f9944f-428t8" Jul 30 19:56:50.186206 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:50.186187 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/080673da-c4bc-4568-bcaa-07e0bea5ad87-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-57cec089-57b78ffbcb-lfhcx\" (UID: \"080673da-c4bc-4568-bcaa-07e0bea5ad87\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-57cec089-57b78ffbcb-lfhcx" Jul 30 19:56:50.195589 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:50.195547 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pspsj\" (UniqueName: \"kubernetes.io/projected/080673da-c4bc-4568-bcaa-07e0bea5ad87-kube-api-access-pspsj\") pod \"payload-pre-processing-e2e-migrate-57cec089-57b78ffbcb-lfhcx\" (UID: \"080673da-c4bc-4568-bcaa-07e0bea5ad87\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-57cec089-57b78ffbcb-lfhcx" Jul 30 19:56:50.286151 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:50.286125 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtlmx\" (UniqueName: \"kubernetes.io/projected/8b5fd0dc-5af2-41d1-9ca9-4b21bca3cdfb-kube-api-access-xtlmx\") pod \"payload-processing-e2e-migrate-57cec089-7dc4f9944f-428t8\" (UID: \"8b5fd0dc-5af2-41d1-9ca9-4b21bca3cdfb\") " pod="openshift-ingress/payload-processing-e2e-migrate-57cec089-7dc4f9944f-428t8" Jul 30 19:56:50.286253 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:50.286162 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b5fd0dc-5af2-41d1-9ca9-4b21bca3cdfb-plugins-config-volume\") pod \"payload-processing-e2e-migrate-57cec089-7dc4f9944f-428t8\" (UID: \"8b5fd0dc-5af2-41d1-9ca9-4b21bca3cdfb\") " pod="openshift-ingress/payload-processing-e2e-migrate-57cec089-7dc4f9944f-428t8" Jul 30 19:56:50.286762 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:50.286746 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b5fd0dc-5af2-41d1-9ca9-4b21bca3cdfb-plugins-config-volume\") pod \"payload-processing-e2e-migrate-57cec089-7dc4f9944f-428t8\" (UID: \"8b5fd0dc-5af2-41d1-9ca9-4b21bca3cdfb\") " pod="openshift-ingress/payload-processing-e2e-migrate-57cec089-7dc4f9944f-428t8" Jul 30 19:56:50.293078 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:50.293056 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtlmx\" (UniqueName: \"kubernetes.io/projected/8b5fd0dc-5af2-41d1-9ca9-4b21bca3cdfb-kube-api-access-xtlmx\") pod \"payload-processing-e2e-migrate-57cec089-7dc4f9944f-428t8\" (UID: \"8b5fd0dc-5af2-41d1-9ca9-4b21bca3cdfb\") " pod="openshift-ingress/payload-processing-e2e-migrate-57cec089-7dc4f9944f-428t8" Jul 30 19:56:50.376510 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:50.376490 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-57cec089-57b78ffbcb-lfhcx" Jul 30 19:56:50.418026 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:50.417991 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-57cec089-7dc4f9944f-428t8" Jul 30 19:56:50.518089 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:50.518049 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-57cec089-57b78ffbcb-lfhcx"] Jul 30 19:56:50.521520 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:56:50.521493 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080673da_c4bc_4568_bcaa_07e0bea5ad87.slice/crio-1f0491e5ca3b60f4ed63baba9b5888f71376598fa76c7ca050a792254e91ff37 WatchSource:0}: Error finding container 1f0491e5ca3b60f4ed63baba9b5888f71376598fa76c7ca050a792254e91ff37: Status 404 returned error can't find the container with id 1f0491e5ca3b60f4ed63baba9b5888f71376598fa76c7ca050a792254e91ff37 Jul 30 19:56:50.757516 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:50.757420 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-57cec089-57b78ffbcb-lfhcx" event={"ID":"080673da-c4bc-4568-bcaa-07e0bea5ad87","Type":"ContainerStarted","Data":"aa0ecd7f84b3c8771292809c60e11f6f65ef6c9445942b3732b32515792378be"} Jul 30 19:56:50.757516 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:50.757493 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-57cec089-57b78ffbcb-lfhcx" event={"ID":"080673da-c4bc-4568-bcaa-07e0bea5ad87","Type":"ContainerStarted","Data":"1f0491e5ca3b60f4ed63baba9b5888f71376598fa76c7ca050a792254e91ff37"} Jul 30 19:56:50.757725 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:50.757621 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-57cec089-57b78ffbcb-lfhcx" Jul 30 19:56:50.773506 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:50.773461 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-57cec089-57b78ffbcb-lfhcx" podStartSLOduration=0.773446028 podStartE2EDuration="773.446028ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 19:56:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:56:50.772068035 +0000 UTC m=+1932.991626952" watchObservedRunningTime="2026-07-30 19:56:50.773446028 +0000 UTC m=+1932.993004940" Jul 30 19:56:50.778393 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:50.778360 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-57cec089-7dc4f9944f-428t8"] Jul 30 19:56:50.780034 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:56:50.780007 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b5fd0dc_5af2_41d1_9ca9_4b21bca3cdfb.slice/crio-34ff34bfe0c9eaf7e44acd5fae2af5f1f11a171a60cfe71f53d1183cb509dd15 WatchSource:0}: Error finding container 34ff34bfe0c9eaf7e44acd5fae2af5f1f11a171a60cfe71f53d1183cb509dd15: Status 404 returned error can't find the container with id 34ff34bfe0c9eaf7e44acd5fae2af5f1f11a171a60cfe71f53d1183cb509dd15 Jul 30 19:56:51.765077 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:51.765036 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-57cec089-7dc4f9944f-428t8" event={"ID":"8b5fd0dc-5af2-41d1-9ca9-4b21bca3cdfb","Type":"ContainerStarted","Data":"8de03ad2311a00513644b35f92e6f6adc87c818d5602f6661dfd81e8512a5954"} Jul 30 19:56:51.765619 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:51.765099 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-57cec089-7dc4f9944f-428t8" event={"ID":"8b5fd0dc-5af2-41d1-9ca9-4b21bca3cdfb","Type":"ContainerStarted","Data":"34ff34bfe0c9eaf7e44acd5fae2af5f1f11a171a60cfe71f53d1183cb509dd15"} Jul 30 19:56:51.765619 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:51.765132 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-57cec089-7dc4f9944f-428t8" Jul 30 19:56:51.780432 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:56:51.780363 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-57cec089-7dc4f9944f-428t8" podStartSLOduration=1.7803490100000001 podStartE2EDuration="1.78034901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 19:56:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:56:51.780019992 +0000 UTC m=+1933.999578905" watchObservedRunningTime="2026-07-30 19:56:51.78034901 +0000 UTC m=+1933.999907922" Jul 30 19:57:01.767243 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:01.767207 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-57cec089-57b78ffbcb-lfhcx" Jul 30 19:57:02.772291 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:02.772264 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-57cec089-7dc4f9944f-428t8" Jul 30 19:57:20.890643 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:20.890608 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-57cec089-7dc4f9944f-428t8"] Jul 30 19:57:20.891159 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:20.890840 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-57cec089-7dc4f9944f-428t8" podUID="8b5fd0dc-5af2-41d1-9ca9-4b21bca3cdfb" containerName="payload-processing" containerID="cri-o://8de03ad2311a00513644b35f92e6f6adc87c818d5602f6661dfd81e8512a5954" gracePeriod=30 Jul 30 19:57:20.895731 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:20.895694 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-57cec089-57b78ffbcb-lfhcx"] Jul 30 19:57:20.896154 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:20.896103 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-57cec089-57b78ffbcb-lfhcx" podUID="080673da-c4bc-4568-bcaa-07e0bea5ad87" containerName="payload-pre-processing" containerID="cri-o://aa0ecd7f84b3c8771292809c60e11f6f65ef6c9445942b3732b32515792378be" gracePeriod=30 Jul 30 19:57:21.188625 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:21.188602 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-57cec089-7dc4f9944f-428t8" Jul 30 19:57:21.192834 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:21.192813 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-57cec089-57b78ffbcb-lfhcx" Jul 30 19:57:21.356252 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:21.356212 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xtlmx\" (UniqueName: \"kubernetes.io/projected/8b5fd0dc-5af2-41d1-9ca9-4b21bca3cdfb-kube-api-access-xtlmx\") pod \"8b5fd0dc-5af2-41d1-9ca9-4b21bca3cdfb\" (UID: \"8b5fd0dc-5af2-41d1-9ca9-4b21bca3cdfb\") " Jul 30 19:57:21.356252 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:21.356259 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b5fd0dc-5af2-41d1-9ca9-4b21bca3cdfb-plugins-config-volume\") pod \"8b5fd0dc-5af2-41d1-9ca9-4b21bca3cdfb\" (UID: \"8b5fd0dc-5af2-41d1-9ca9-4b21bca3cdfb\") " Jul 30 19:57:21.356525 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:21.356287 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/080673da-c4bc-4568-bcaa-07e0bea5ad87-plugins-config-volume\") pod \"080673da-c4bc-4568-bcaa-07e0bea5ad87\" (UID: \"080673da-c4bc-4568-bcaa-07e0bea5ad87\") " Jul 30 19:57:21.356525 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:21.356473 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pspsj\" (UniqueName: \"kubernetes.io/projected/080673da-c4bc-4568-bcaa-07e0bea5ad87-kube-api-access-pspsj\") pod \"080673da-c4bc-4568-bcaa-07e0bea5ad87\" (UID: \"080673da-c4bc-4568-bcaa-07e0bea5ad87\") " Jul 30 19:57:21.356671 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:21.356646 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b5fd0dc-5af2-41d1-9ca9-4b21bca3cdfb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8b5fd0dc-5af2-41d1-9ca9-4b21bca3cdfb" (UID: "8b5fd0dc-5af2-41d1-9ca9-4b21bca3cdfb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 19:57:21.356802 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:21.356683 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/080673da-c4bc-4568-bcaa-07e0bea5ad87-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "080673da-c4bc-4568-bcaa-07e0bea5ad87" (UID: "080673da-c4bc-4568-bcaa-07e0bea5ad87"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 19:57:21.356854 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:21.356830 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b5fd0dc-5af2-41d1-9ca9-4b21bca3cdfb-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:57:21.356854 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:21.356845 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/080673da-c4bc-4568-bcaa-07e0bea5ad87-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:57:21.358321 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:21.358301 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b5fd0dc-5af2-41d1-9ca9-4b21bca3cdfb-kube-api-access-xtlmx" (OuterVolumeSpecName: "kube-api-access-xtlmx") pod "8b5fd0dc-5af2-41d1-9ca9-4b21bca3cdfb" (UID: "8b5fd0dc-5af2-41d1-9ca9-4b21bca3cdfb"). InnerVolumeSpecName "kube-api-access-xtlmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 19:57:21.358321 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:21.358313 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/080673da-c4bc-4568-bcaa-07e0bea5ad87-kube-api-access-pspsj" (OuterVolumeSpecName: "kube-api-access-pspsj") pod "080673da-c4bc-4568-bcaa-07e0bea5ad87" (UID: "080673da-c4bc-4568-bcaa-07e0bea5ad87"). InnerVolumeSpecName "kube-api-access-pspsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 19:57:21.457891 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:21.457798 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtlmx\" (UniqueName: \"kubernetes.io/projected/8b5fd0dc-5af2-41d1-9ca9-4b21bca3cdfb-kube-api-access-xtlmx\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:57:21.457891 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:21.457836 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pspsj\" (UniqueName: \"kubernetes.io/projected/080673da-c4bc-4568-bcaa-07e0bea5ad87-kube-api-access-pspsj\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:57:21.929217 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:21.929182 2556 generic.go:356] "Generic (PLEG): container finished" podID="080673da-c4bc-4568-bcaa-07e0bea5ad87" containerID="aa0ecd7f84b3c8771292809c60e11f6f65ef6c9445942b3732b32515792378be" exitCode=0 Jul 30 19:57:21.929726 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:21.929233 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-57cec089-57b78ffbcb-lfhcx" Jul 30 19:57:21.929726 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:21.929278 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-57cec089-57b78ffbcb-lfhcx" event={"ID":"080673da-c4bc-4568-bcaa-07e0bea5ad87","Type":"ContainerDied","Data":"aa0ecd7f84b3c8771292809c60e11f6f65ef6c9445942b3732b32515792378be"} Jul 30 19:57:21.929726 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:21.929340 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-57cec089-57b78ffbcb-lfhcx" event={"ID":"080673da-c4bc-4568-bcaa-07e0bea5ad87","Type":"ContainerDied","Data":"1f0491e5ca3b60f4ed63baba9b5888f71376598fa76c7ca050a792254e91ff37"} Jul 30 19:57:21.929726 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:21.929374 2556 scope.go:117] "RemoveContainer" containerID="aa0ecd7f84b3c8771292809c60e11f6f65ef6c9445942b3732b32515792378be" Jul 30 19:57:21.931307 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:21.931057 2556 generic.go:356] "Generic (PLEG): container finished" podID="8b5fd0dc-5af2-41d1-9ca9-4b21bca3cdfb" containerID="8de03ad2311a00513644b35f92e6f6adc87c818d5602f6661dfd81e8512a5954" exitCode=0 Jul 30 19:57:21.931307 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:21.931148 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-57cec089-7dc4f9944f-428t8" Jul 30 19:57:21.931307 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:21.931156 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-57cec089-7dc4f9944f-428t8" event={"ID":"8b5fd0dc-5af2-41d1-9ca9-4b21bca3cdfb","Type":"ContainerDied","Data":"8de03ad2311a00513644b35f92e6f6adc87c818d5602f6661dfd81e8512a5954"} Jul 30 19:57:21.931307 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:21.931197 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-57cec089-7dc4f9944f-428t8" event={"ID":"8b5fd0dc-5af2-41d1-9ca9-4b21bca3cdfb","Type":"ContainerDied","Data":"34ff34bfe0c9eaf7e44acd5fae2af5f1f11a171a60cfe71f53d1183cb509dd15"} Jul 30 19:57:21.940459 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:21.940164 2556 scope.go:117] "RemoveContainer" containerID="aa0ecd7f84b3c8771292809c60e11f6f65ef6c9445942b3732b32515792378be" Jul 30 19:57:21.940706 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:57:21.940540 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa0ecd7f84b3c8771292809c60e11f6f65ef6c9445942b3732b32515792378be\": container with ID starting with aa0ecd7f84b3c8771292809c60e11f6f65ef6c9445942b3732b32515792378be not found: ID does not exist" containerID="aa0ecd7f84b3c8771292809c60e11f6f65ef6c9445942b3732b32515792378be" Jul 30 19:57:21.940706 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:21.940579 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa0ecd7f84b3c8771292809c60e11f6f65ef6c9445942b3732b32515792378be"} err="failed to get container status \"aa0ecd7f84b3c8771292809c60e11f6f65ef6c9445942b3732b32515792378be\": rpc error: code = NotFound desc = could not find container \"aa0ecd7f84b3c8771292809c60e11f6f65ef6c9445942b3732b32515792378be\": container with ID starting with aa0ecd7f84b3c8771292809c60e11f6f65ef6c9445942b3732b32515792378be not found: ID does not exist" Jul 30 19:57:21.940706 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:21.940604 2556 scope.go:117] "RemoveContainer" containerID="8de03ad2311a00513644b35f92e6f6adc87c818d5602f6661dfd81e8512a5954" Jul 30 19:57:21.952433 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:21.952413 2556 scope.go:117] "RemoveContainer" containerID="8de03ad2311a00513644b35f92e6f6adc87c818d5602f6661dfd81e8512a5954" Jul 30 19:57:21.952764 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:57:21.952742 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8de03ad2311a00513644b35f92e6f6adc87c818d5602f6661dfd81e8512a5954\": container with ID starting with 8de03ad2311a00513644b35f92e6f6adc87c818d5602f6661dfd81e8512a5954 not found: ID does not exist" containerID="8de03ad2311a00513644b35f92e6f6adc87c818d5602f6661dfd81e8512a5954" Jul 30 19:57:21.952849 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:21.952770 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8de03ad2311a00513644b35f92e6f6adc87c818d5602f6661dfd81e8512a5954"} err="failed to get container status \"8de03ad2311a00513644b35f92e6f6adc87c818d5602f6661dfd81e8512a5954\": rpc error: code = NotFound desc = could not find container \"8de03ad2311a00513644b35f92e6f6adc87c818d5602f6661dfd81e8512a5954\": container with ID starting with 8de03ad2311a00513644b35f92e6f6adc87c818d5602f6661dfd81e8512a5954 not found: ID does not exist" Jul 30 19:57:21.962970 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:21.962950 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-57cec089-57b78ffbcb-lfhcx"] Jul 30 19:57:21.970844 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:21.970820 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-57cec089-57b78ffbcb-lfhcx"] Jul 30 19:57:21.980311 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:21.980148 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-57cec089-7dc4f9944f-428t8"] Jul 30 19:57:21.982195 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:21.982174 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-57cec089-7dc4f9944f-428t8"] Jul 30 19:57:22.324119 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:22.324035 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="080673da-c4bc-4568-bcaa-07e0bea5ad87" path="/var/lib/kubelet/pods/080673da-c4bc-4568-bcaa-07e0bea5ad87/volumes" Jul 30 19:57:22.324464 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:22.324449 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b5fd0dc-5af2-41d1-9ca9-4b21bca3cdfb" path="/var/lib/kubelet/pods/8b5fd0dc-5af2-41d1-9ca9-4b21bca3cdfb/volumes" Jul 30 19:57:25.836484 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:25.836451 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a76d0719-649cb8bcd4-4qm2t"] Jul 30 19:57:25.837171 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:25.837054 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="080673da-c4bc-4568-bcaa-07e0bea5ad87" containerName="payload-pre-processing" Jul 30 19:57:25.837171 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:25.837079 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="080673da-c4bc-4568-bcaa-07e0bea5ad87" containerName="payload-pre-processing" Jul 30 19:57:25.837171 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:25.837130 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b5fd0dc-5af2-41d1-9ca9-4b21bca3cdfb" containerName="payload-processing" Jul 30 19:57:25.837171 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:25.837139 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b5fd0dc-5af2-41d1-9ca9-4b21bca3cdfb" containerName="payload-processing" Jul 30 19:57:25.837504 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:25.837255 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="080673da-c4bc-4568-bcaa-07e0bea5ad87" containerName="payload-pre-processing" Jul 30 19:57:25.837504 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:25.837272 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b5fd0dc-5af2-41d1-9ca9-4b21bca3cdfb" containerName="payload-processing" Jul 30 19:57:25.840573 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:25.840554 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a76d0719-649cb8bcd4-4qm2t" Jul 30 19:57:25.842967 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:25.842951 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a76d0719\"" Jul 30 19:57:25.851228 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:25.851201 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a76d0719-649cb8bcd4-4qm2t"] Jul 30 19:57:25.885727 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:25.885702 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a76d0719-6f58cd5c8-zqpvh"] Jul 30 19:57:25.889117 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:25.889093 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a76d0719-6f58cd5c8-zqpvh" Jul 30 19:57:25.892079 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:25.892054 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a76d0719-dockercfg-hgdnv\"" Jul 30 19:57:25.902440 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:25.902406 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a76d0719-6f58cd5c8-zqpvh"] Jul 30 19:57:26.002833 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:26.002796 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k75wc\" (UniqueName: \"kubernetes.io/projected/2741d4ef-4c98-4bd3-b7ec-f075f460898e-kube-api-access-k75wc\") pod \"payload-pre-processing-e2e-ait-a76d0719-649cb8bcd4-4qm2t\" (UID: \"2741d4ef-4c98-4bd3-b7ec-f075f460898e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a76d0719-649cb8bcd4-4qm2t" Jul 30 19:57:26.003018 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:26.002847 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2741d4ef-4c98-4bd3-b7ec-f075f460898e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a76d0719-649cb8bcd4-4qm2t\" (UID: \"2741d4ef-4c98-4bd3-b7ec-f075f460898e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a76d0719-649cb8bcd4-4qm2t" Jul 30 19:57:26.003018 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:26.002962 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57cd6c97-960f-46b4-96a2-3e31607d2fe9-plugins-config-volume\") pod \"payload-processing-e2e-ait-a76d0719-6f58cd5c8-zqpvh\" (UID: \"57cd6c97-960f-46b4-96a2-3e31607d2fe9\") " pod="openshift-ingress/payload-processing-e2e-ait-a76d0719-6f58cd5c8-zqpvh" Jul 30 19:57:26.003119 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:26.003023 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4452g\" (UniqueName: \"kubernetes.io/projected/57cd6c97-960f-46b4-96a2-3e31607d2fe9-kube-api-access-4452g\") pod \"payload-processing-e2e-ait-a76d0719-6f58cd5c8-zqpvh\" (UID: \"57cd6c97-960f-46b4-96a2-3e31607d2fe9\") " pod="openshift-ingress/payload-processing-e2e-ait-a76d0719-6f58cd5c8-zqpvh" Jul 30 19:57:26.103926 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:26.103839 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2741d4ef-4c98-4bd3-b7ec-f075f460898e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a76d0719-649cb8bcd4-4qm2t\" (UID: \"2741d4ef-4c98-4bd3-b7ec-f075f460898e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a76d0719-649cb8bcd4-4qm2t" Jul 30 19:57:26.104085 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:26.103934 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57cd6c97-960f-46b4-96a2-3e31607d2fe9-plugins-config-volume\") pod \"payload-processing-e2e-ait-a76d0719-6f58cd5c8-zqpvh\" (UID: \"57cd6c97-960f-46b4-96a2-3e31607d2fe9\") " pod="openshift-ingress/payload-processing-e2e-ait-a76d0719-6f58cd5c8-zqpvh" Jul 30 19:57:26.104085 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:26.103986 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4452g\" (UniqueName: \"kubernetes.io/projected/57cd6c97-960f-46b4-96a2-3e31607d2fe9-kube-api-access-4452g\") pod \"payload-processing-e2e-ait-a76d0719-6f58cd5c8-zqpvh\" (UID: \"57cd6c97-960f-46b4-96a2-3e31607d2fe9\") " pod="openshift-ingress/payload-processing-e2e-ait-a76d0719-6f58cd5c8-zqpvh" Jul 30 19:57:26.104085 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:26.104051 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k75wc\" (UniqueName: \"kubernetes.io/projected/2741d4ef-4c98-4bd3-b7ec-f075f460898e-kube-api-access-k75wc\") pod \"payload-pre-processing-e2e-ait-a76d0719-649cb8bcd4-4qm2t\" (UID: \"2741d4ef-4c98-4bd3-b7ec-f075f460898e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a76d0719-649cb8bcd4-4qm2t" Jul 30 19:57:26.104686 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:26.104660 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57cd6c97-960f-46b4-96a2-3e31607d2fe9-plugins-config-volume\") pod \"payload-processing-e2e-ait-a76d0719-6f58cd5c8-zqpvh\" (UID: \"57cd6c97-960f-46b4-96a2-3e31607d2fe9\") " pod="openshift-ingress/payload-processing-e2e-ait-a76d0719-6f58cd5c8-zqpvh" Jul 30 19:57:26.104816 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:26.104697 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2741d4ef-4c98-4bd3-b7ec-f075f460898e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a76d0719-649cb8bcd4-4qm2t\" (UID: \"2741d4ef-4c98-4bd3-b7ec-f075f460898e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a76d0719-649cb8bcd4-4qm2t" Jul 30 19:57:26.111300 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:26.111273 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k75wc\" (UniqueName: \"kubernetes.io/projected/2741d4ef-4c98-4bd3-b7ec-f075f460898e-kube-api-access-k75wc\") pod \"payload-pre-processing-e2e-ait-a76d0719-649cb8bcd4-4qm2t\" (UID: \"2741d4ef-4c98-4bd3-b7ec-f075f460898e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a76d0719-649cb8bcd4-4qm2t" Jul 30 19:57:26.111455 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:26.111358 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4452g\" (UniqueName: \"kubernetes.io/projected/57cd6c97-960f-46b4-96a2-3e31607d2fe9-kube-api-access-4452g\") pod \"payload-processing-e2e-ait-a76d0719-6f58cd5c8-zqpvh\" (UID: \"57cd6c97-960f-46b4-96a2-3e31607d2fe9\") " pod="openshift-ingress/payload-processing-e2e-ait-a76d0719-6f58cd5c8-zqpvh" Jul 30 19:57:26.152086 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:26.152062 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a76d0719-649cb8bcd4-4qm2t" Jul 30 19:57:26.200547 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:26.200519 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a76d0719-6f58cd5c8-zqpvh" Jul 30 19:57:26.289676 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:26.289650 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a76d0719-649cb8bcd4-4qm2t"] Jul 30 19:57:26.291761 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:57:26.291738 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2741d4ef_4c98_4bd3_b7ec_f075f460898e.slice/crio-01dd5601734b375df9e4f871a24a26259247da06b6e4d67efe1571a003616f1a WatchSource:0}: Error finding container 01dd5601734b375df9e4f871a24a26259247da06b6e4d67efe1571a003616f1a: Status 404 returned error can't find the container with id 01dd5601734b375df9e4f871a24a26259247da06b6e4d67efe1571a003616f1a Jul 30 19:57:26.346630 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:26.346608 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a76d0719-6f58cd5c8-zqpvh"] Jul 30 19:57:26.348237 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:57:26.348213 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57cd6c97_960f_46b4_96a2_3e31607d2fe9.slice/crio-cc1daa0fe6c469044016149bea324e6a0e55211778f56f341b29bf8dc975adbb WatchSource:0}: Error finding container cc1daa0fe6c469044016149bea324e6a0e55211778f56f341b29bf8dc975adbb: Status 404 returned error can't find the container with id cc1daa0fe6c469044016149bea324e6a0e55211778f56f341b29bf8dc975adbb Jul 30 19:57:26.959994 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:26.959960 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a76d0719-649cb8bcd4-4qm2t" event={"ID":"2741d4ef-4c98-4bd3-b7ec-f075f460898e","Type":"ContainerStarted","Data":"68158822e4bd49e077fc7a35de9d0350dcf63f6d8cd8853c6284f62593444546"} Jul 30 19:57:26.960446 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:26.960014 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a76d0719-649cb8bcd4-4qm2t" event={"ID":"2741d4ef-4c98-4bd3-b7ec-f075f460898e","Type":"ContainerStarted","Data":"01dd5601734b375df9e4f871a24a26259247da06b6e4d67efe1571a003616f1a"} Jul 30 19:57:26.960446 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:26.960048 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a76d0719-649cb8bcd4-4qm2t" Jul 30 19:57:26.961529 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:26.961506 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a76d0719-6f58cd5c8-zqpvh" event={"ID":"57cd6c97-960f-46b4-96a2-3e31607d2fe9","Type":"ContainerStarted","Data":"21b8bc9752d0700bc26ece22d377556528f74565459291215f8cf048ad336f31"} Jul 30 19:57:26.961636 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:26.961556 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a76d0719-6f58cd5c8-zqpvh" Jul 30 19:57:26.961636 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:26.961573 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a76d0719-6f58cd5c8-zqpvh" event={"ID":"57cd6c97-960f-46b4-96a2-3e31607d2fe9","Type":"ContainerStarted","Data":"cc1daa0fe6c469044016149bea324e6a0e55211778f56f341b29bf8dc975adbb"} Jul 30 19:57:26.978399 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:26.978348 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-a76d0719-649cb8bcd4-4qm2t" podStartSLOduration=1.9783337429999999 podStartE2EDuration="1.978333743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 19:57:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:57:26.976943915 +0000 UTC m=+1969.196502827" watchObservedRunningTime="2026-07-30 19:57:26.978333743 +0000 UTC m=+1969.197892657" Jul 30 19:57:37.969593 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:37.969500 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a76d0719-6f58cd5c8-zqpvh" Jul 30 19:57:37.969593 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:37.969583 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a76d0719-649cb8bcd4-4qm2t" Jul 30 19:57:37.985073 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:37.985025 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a76d0719-6f58cd5c8-zqpvh" podStartSLOduration=12.985011933 podStartE2EDuration="12.985011933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 19:57:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:57:26.991472746 +0000 UTC m=+1969.211031661" watchObservedRunningTime="2026-07-30 19:57:37.985011933 +0000 UTC m=+1980.204570846" Jul 30 19:57:55.900273 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:55.900238 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a76d0719-6f58cd5c8-zqpvh"] Jul 30 19:57:55.900740 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:55.900502 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a76d0719-6f58cd5c8-zqpvh" podUID="57cd6c97-960f-46b4-96a2-3e31607d2fe9" containerName="payload-processing" containerID="cri-o://21b8bc9752d0700bc26ece22d377556528f74565459291215f8cf048ad336f31" gracePeriod=30 Jul 30 19:57:55.911320 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:55.911294 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a76d0719-649cb8bcd4-4qm2t"] Jul 30 19:57:55.911731 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:55.911683 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-a76d0719-649cb8bcd4-4qm2t" podUID="2741d4ef-4c98-4bd3-b7ec-f075f460898e" containerName="payload-pre-processing" containerID="cri-o://68158822e4bd49e077fc7a35de9d0350dcf63f6d8cd8853c6284f62593444546" gracePeriod=30 Jul 30 19:57:56.119575 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:56.119541 2556 generic.go:356] "Generic (PLEG): container finished" podID="2741d4ef-4c98-4bd3-b7ec-f075f460898e" containerID="68158822e4bd49e077fc7a35de9d0350dcf63f6d8cd8853c6284f62593444546" exitCode=0 Jul 30 19:57:56.119761 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:56.119619 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a76d0719-649cb8bcd4-4qm2t" event={"ID":"2741d4ef-4c98-4bd3-b7ec-f075f460898e","Type":"ContainerDied","Data":"68158822e4bd49e077fc7a35de9d0350dcf63f6d8cd8853c6284f62593444546"} Jul 30 19:57:56.121367 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:56.121236 2556 generic.go:356] "Generic (PLEG): container finished" podID="57cd6c97-960f-46b4-96a2-3e31607d2fe9" containerID="21b8bc9752d0700bc26ece22d377556528f74565459291215f8cf048ad336f31" exitCode=0 Jul 30 19:57:56.121367 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:56.121268 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a76d0719-6f58cd5c8-zqpvh" event={"ID":"57cd6c97-960f-46b4-96a2-3e31607d2fe9","Type":"ContainerDied","Data":"21b8bc9752d0700bc26ece22d377556528f74565459291215f8cf048ad336f31"} Jul 30 19:57:56.162223 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:56.162169 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a76d0719-6f58cd5c8-zqpvh" Jul 30 19:57:56.191853 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:56.191825 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a76d0719-649cb8bcd4-4qm2t" Jul 30 19:57:56.253503 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:56.253481 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k75wc\" (UniqueName: \"kubernetes.io/projected/2741d4ef-4c98-4bd3-b7ec-f075f460898e-kube-api-access-k75wc\") pod \"2741d4ef-4c98-4bd3-b7ec-f075f460898e\" (UID: \"2741d4ef-4c98-4bd3-b7ec-f075f460898e\") " Jul 30 19:57:56.253613 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:56.253535 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57cd6c97-960f-46b4-96a2-3e31607d2fe9-plugins-config-volume\") pod \"57cd6c97-960f-46b4-96a2-3e31607d2fe9\" (UID: \"57cd6c97-960f-46b4-96a2-3e31607d2fe9\") " Jul 30 19:57:56.253613 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:56.253568 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2741d4ef-4c98-4bd3-b7ec-f075f460898e-plugins-config-volume\") pod \"2741d4ef-4c98-4bd3-b7ec-f075f460898e\" (UID: \"2741d4ef-4c98-4bd3-b7ec-f075f460898e\") " Jul 30 19:57:56.253613 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:56.253605 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4452g\" (UniqueName: \"kubernetes.io/projected/57cd6c97-960f-46b4-96a2-3e31607d2fe9-kube-api-access-4452g\") pod \"57cd6c97-960f-46b4-96a2-3e31607d2fe9\" (UID: \"57cd6c97-960f-46b4-96a2-3e31607d2fe9\") " Jul 30 19:57:56.253974 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:56.253941 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57cd6c97-960f-46b4-96a2-3e31607d2fe9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "57cd6c97-960f-46b4-96a2-3e31607d2fe9" (UID: "57cd6c97-960f-46b4-96a2-3e31607d2fe9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 19:57:56.253974 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:56.253956 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2741d4ef-4c98-4bd3-b7ec-f075f460898e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2741d4ef-4c98-4bd3-b7ec-f075f460898e" (UID: "2741d4ef-4c98-4bd3-b7ec-f075f460898e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 19:57:56.255513 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:56.255485 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2741d4ef-4c98-4bd3-b7ec-f075f460898e-kube-api-access-k75wc" (OuterVolumeSpecName: "kube-api-access-k75wc") pod "2741d4ef-4c98-4bd3-b7ec-f075f460898e" (UID: "2741d4ef-4c98-4bd3-b7ec-f075f460898e"). InnerVolumeSpecName "kube-api-access-k75wc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 19:57:56.255649 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:56.255630 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57cd6c97-960f-46b4-96a2-3e31607d2fe9-kube-api-access-4452g" (OuterVolumeSpecName: "kube-api-access-4452g") pod "57cd6c97-960f-46b4-96a2-3e31607d2fe9" (UID: "57cd6c97-960f-46b4-96a2-3e31607d2fe9"). InnerVolumeSpecName "kube-api-access-4452g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 19:57:56.355182 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:56.355159 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k75wc\" (UniqueName: \"kubernetes.io/projected/2741d4ef-4c98-4bd3-b7ec-f075f460898e-kube-api-access-k75wc\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:57:56.355182 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:56.355180 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57cd6c97-960f-46b4-96a2-3e31607d2fe9-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:57:56.355307 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:56.355190 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2741d4ef-4c98-4bd3-b7ec-f075f460898e-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:57:56.355307 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:56.355199 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4452g\" (UniqueName: \"kubernetes.io/projected/57cd6c97-960f-46b4-96a2-3e31607d2fe9-kube-api-access-4452g\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:57:57.127726 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:57.127699 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a76d0719-6f58cd5c8-zqpvh" Jul 30 19:57:57.127726 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:57.127707 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a76d0719-6f58cd5c8-zqpvh" event={"ID":"57cd6c97-960f-46b4-96a2-3e31607d2fe9","Type":"ContainerDied","Data":"cc1daa0fe6c469044016149bea324e6a0e55211778f56f341b29bf8dc975adbb"} Jul 30 19:57:57.128206 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:57.127793 2556 scope.go:117] "RemoveContainer" containerID="21b8bc9752d0700bc26ece22d377556528f74565459291215f8cf048ad336f31" Jul 30 19:57:57.129165 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:57.129148 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a76d0719-649cb8bcd4-4qm2t" Jul 30 19:57:57.129251 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:57.129208 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a76d0719-649cb8bcd4-4qm2t" event={"ID":"2741d4ef-4c98-4bd3-b7ec-f075f460898e","Type":"ContainerDied","Data":"01dd5601734b375df9e4f871a24a26259247da06b6e4d67efe1571a003616f1a"} Jul 30 19:57:57.136775 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:57.136761 2556 scope.go:117] "RemoveContainer" containerID="68158822e4bd49e077fc7a35de9d0350dcf63f6d8cd8853c6284f62593444546" Jul 30 19:57:57.145585 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:57.145568 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a76d0719-649cb8bcd4-4qm2t"] Jul 30 19:57:57.150095 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:57.150076 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a76d0719-649cb8bcd4-4qm2t"] Jul 30 19:57:57.159525 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:57.159505 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a76d0719-6f58cd5c8-zqpvh"] Jul 30 19:57:57.163240 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:57.163218 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a76d0719-6f58cd5c8-zqpvh"] Jul 30 19:57:58.323722 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:58.323693 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2741d4ef-4c98-4bd3-b7ec-f075f460898e" path="/var/lib/kubelet/pods/2741d4ef-4c98-4bd3-b7ec-f075f460898e/volumes" Jul 30 19:57:58.324093 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:57:58.324033 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57cd6c97-960f-46b4-96a2-3e31607d2fe9" path="/var/lib/kubelet/pods/57cd6c97-960f-46b4-96a2-3e31607d2fe9/volumes" Jul 30 19:58:02.155262 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:02.155233 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b83b31b4-7857766488-8949s"] Jul 30 19:58:02.155712 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:02.155698 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2741d4ef-4c98-4bd3-b7ec-f075f460898e" containerName="payload-pre-processing" Jul 30 19:58:02.155758 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:02.155713 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="2741d4ef-4c98-4bd3-b7ec-f075f460898e" containerName="payload-pre-processing" Jul 30 19:58:02.155758 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:02.155739 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57cd6c97-960f-46b4-96a2-3e31607d2fe9" containerName="payload-processing" Jul 30 19:58:02.155758 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:02.155745 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="57cd6c97-960f-46b4-96a2-3e31607d2fe9" containerName="payload-processing" Jul 30 19:58:02.155855 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:02.155818 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="2741d4ef-4c98-4bd3-b7ec-f075f460898e" containerName="payload-pre-processing" Jul 30 19:58:02.155855 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:02.155826 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="57cd6c97-960f-46b4-96a2-3e31607d2fe9" containerName="payload-processing" Jul 30 19:58:02.160165 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:02.160148 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-b83b31b4-7857766488-8949s" Jul 30 19:58:02.162814 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:02.162789 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-b83b31b4\"" Jul 30 19:58:02.162951 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:02.162901 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-b83b31b4-dockercfg-gq9pm\"" Jul 30 19:58:02.168478 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:02.168230 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b83b31b4-7857766488-8949s"] Jul 30 19:58:02.192432 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:02.192376 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b83b31b4-7f694d65c8-267bk"] Jul 30 19:58:02.195973 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:02.195955 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-b83b31b4-7f694d65c8-267bk" Jul 30 19:58:02.202470 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:02.202447 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b83b31b4-7f694d65c8-267bk"] Jul 30 19:58:02.306297 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:02.306259 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdf63569-4cbd-40a9-a1e1-0dbf00641da6-plugins-config-volume\") pod \"payload-processing-e2e-derive-b83b31b4-7f694d65c8-267bk\" (UID: \"fdf63569-4cbd-40a9-a1e1-0dbf00641da6\") " pod="openshift-ingress/payload-processing-e2e-derive-b83b31b4-7f694d65c8-267bk" Jul 30 19:58:02.306475 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:02.306348 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/376f0e16-fa4e-492d-8a4a-9be579d96f65-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-b83b31b4-7857766488-8949s\" (UID: \"376f0e16-fa4e-492d-8a4a-9be579d96f65\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b83b31b4-7857766488-8949s" Jul 30 19:58:02.306475 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:02.306376 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs8rx\" (UniqueName: \"kubernetes.io/projected/fdf63569-4cbd-40a9-a1e1-0dbf00641da6-kube-api-access-hs8rx\") pod \"payload-processing-e2e-derive-b83b31b4-7f694d65c8-267bk\" (UID: \"fdf63569-4cbd-40a9-a1e1-0dbf00641da6\") " pod="openshift-ingress/payload-processing-e2e-derive-b83b31b4-7f694d65c8-267bk" Jul 30 19:58:02.306475 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:02.306431 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5th2w\" (UniqueName: \"kubernetes.io/projected/376f0e16-fa4e-492d-8a4a-9be579d96f65-kube-api-access-5th2w\") pod \"payload-pre-processing-e2e-derive-b83b31b4-7857766488-8949s\" (UID: \"376f0e16-fa4e-492d-8a4a-9be579d96f65\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b83b31b4-7857766488-8949s" Jul 30 19:58:02.407423 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:02.407327 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdf63569-4cbd-40a9-a1e1-0dbf00641da6-plugins-config-volume\") pod \"payload-processing-e2e-derive-b83b31b4-7f694d65c8-267bk\" (UID: \"fdf63569-4cbd-40a9-a1e1-0dbf00641da6\") " pod="openshift-ingress/payload-processing-e2e-derive-b83b31b4-7f694d65c8-267bk" Jul 30 19:58:02.407423 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:02.407407 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/376f0e16-fa4e-492d-8a4a-9be579d96f65-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-b83b31b4-7857766488-8949s\" (UID: \"376f0e16-fa4e-492d-8a4a-9be579d96f65\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b83b31b4-7857766488-8949s" Jul 30 19:58:02.407656 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:02.407441 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs8rx\" (UniqueName: \"kubernetes.io/projected/fdf63569-4cbd-40a9-a1e1-0dbf00641da6-kube-api-access-hs8rx\") pod \"payload-processing-e2e-derive-b83b31b4-7f694d65c8-267bk\" (UID: \"fdf63569-4cbd-40a9-a1e1-0dbf00641da6\") " pod="openshift-ingress/payload-processing-e2e-derive-b83b31b4-7f694d65c8-267bk" Jul 30 19:58:02.407656 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:02.407459 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5th2w\" (UniqueName: \"kubernetes.io/projected/376f0e16-fa4e-492d-8a4a-9be579d96f65-kube-api-access-5th2w\") pod \"payload-pre-processing-e2e-derive-b83b31b4-7857766488-8949s\" (UID: \"376f0e16-fa4e-492d-8a4a-9be579d96f65\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b83b31b4-7857766488-8949s" Jul 30 19:58:02.408174 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:02.408148 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdf63569-4cbd-40a9-a1e1-0dbf00641da6-plugins-config-volume\") pod \"payload-processing-e2e-derive-b83b31b4-7f694d65c8-267bk\" (UID: \"fdf63569-4cbd-40a9-a1e1-0dbf00641da6\") " pod="openshift-ingress/payload-processing-e2e-derive-b83b31b4-7f694d65c8-267bk" Jul 30 19:58:02.408284 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:02.408261 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/376f0e16-fa4e-492d-8a4a-9be579d96f65-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-b83b31b4-7857766488-8949s\" (UID: \"376f0e16-fa4e-492d-8a4a-9be579d96f65\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b83b31b4-7857766488-8949s" Jul 30 19:58:02.415427 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:02.415404 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5th2w\" (UniqueName: \"kubernetes.io/projected/376f0e16-fa4e-492d-8a4a-9be579d96f65-kube-api-access-5th2w\") pod \"payload-pre-processing-e2e-derive-b83b31b4-7857766488-8949s\" (UID: \"376f0e16-fa4e-492d-8a4a-9be579d96f65\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b83b31b4-7857766488-8949s" Jul 30 19:58:02.415549 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:02.415527 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs8rx\" (UniqueName: \"kubernetes.io/projected/fdf63569-4cbd-40a9-a1e1-0dbf00641da6-kube-api-access-hs8rx\") pod \"payload-processing-e2e-derive-b83b31b4-7f694d65c8-267bk\" (UID: \"fdf63569-4cbd-40a9-a1e1-0dbf00641da6\") " pod="openshift-ingress/payload-processing-e2e-derive-b83b31b4-7f694d65c8-267bk" Jul 30 19:58:02.472953 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:02.472930 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-b83b31b4-7857766488-8949s" Jul 30 19:58:02.507090 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:02.507065 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-b83b31b4-7f694d65c8-267bk" Jul 30 19:58:02.828747 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:02.828722 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b83b31b4-7857766488-8949s"] Jul 30 19:58:02.831232 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:58:02.831200 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod376f0e16_fa4e_492d_8a4a_9be579d96f65.slice/crio-e7e2fd9be6ba5c6be14df61d800f352cfcbcca2289b965009788fc2e5b5bcaeb WatchSource:0}: Error finding container e7e2fd9be6ba5c6be14df61d800f352cfcbcca2289b965009788fc2e5b5bcaeb: Status 404 returned error can't find the container with id e7e2fd9be6ba5c6be14df61d800f352cfcbcca2289b965009788fc2e5b5bcaeb Jul 30 19:58:02.855750 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:02.855729 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b83b31b4-7f694d65c8-267bk"] Jul 30 19:58:02.863482 ip-10-0-129-105 kubenswrapper[2556]: W0730 19:58:02.863458 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdf63569_4cbd_40a9_a1e1_0dbf00641da6.slice/crio-de163d6c03e04009774253dd795978c34bc1c495955f5c5657987d2a60c9da2f WatchSource:0}: Error finding container de163d6c03e04009774253dd795978c34bc1c495955f5c5657987d2a60c9da2f: Status 404 returned error can't find the container with id de163d6c03e04009774253dd795978c34bc1c495955f5c5657987d2a60c9da2f Jul 30 19:58:03.162828 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:03.162788 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-b83b31b4-7f694d65c8-267bk" event={"ID":"fdf63569-4cbd-40a9-a1e1-0dbf00641da6","Type":"ContainerStarted","Data":"238b49261ffdce0b51bcc675b1e826c36219c43b272e07bee62432d549cc7508"} Jul 30 19:58:03.163254 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:03.162854 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-b83b31b4-7f694d65c8-267bk" Jul 30 19:58:03.163254 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:03.162877 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-b83b31b4-7f694d65c8-267bk" event={"ID":"fdf63569-4cbd-40a9-a1e1-0dbf00641da6","Type":"ContainerStarted","Data":"de163d6c03e04009774253dd795978c34bc1c495955f5c5657987d2a60c9da2f"} Jul 30 19:58:03.164399 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:03.164360 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-b83b31b4-7857766488-8949s" event={"ID":"376f0e16-fa4e-492d-8a4a-9be579d96f65","Type":"ContainerStarted","Data":"5dd9db673480f638c5a47151f475dc4b2e951628e7cd5b22ca29a6ad7bcf2f42"} Jul 30 19:58:03.164527 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:03.164413 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-b83b31b4-7857766488-8949s" event={"ID":"376f0e16-fa4e-492d-8a4a-9be579d96f65","Type":"ContainerStarted","Data":"e7e2fd9be6ba5c6be14df61d800f352cfcbcca2289b965009788fc2e5b5bcaeb"} Jul 30 19:58:03.164527 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:03.164436 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-b83b31b4-7857766488-8949s" Jul 30 19:58:03.180233 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:03.180192 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-b83b31b4-7f694d65c8-267bk" podStartSLOduration=1.180178929 podStartE2EDuration="1.180178929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 19:58:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:58:03.17877277 +0000 UTC m=+2005.398331685" watchObservedRunningTime="2026-07-30 19:58:03.180178929 +0000 UTC m=+2005.399737893" Jul 30 19:58:03.192965 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:03.192916 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-b83b31b4-7857766488-8949s" podStartSLOduration=1.192898087 podStartE2EDuration="1.192898087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 19:58:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 19:58:03.192620733 +0000 UTC m=+2005.412179657" watchObservedRunningTime="2026-07-30 19:58:03.192898087 +0000 UTC m=+2005.412457001" Jul 30 19:58:14.171395 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:14.171349 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-b83b31b4-7f694d65c8-267bk" Jul 30 19:58:14.171858 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:14.171447 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-b83b31b4-7857766488-8949s" Jul 30 19:58:32.018979 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:32.018946 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b83b31b4-7f694d65c8-267bk"] Jul 30 19:58:32.019831 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:32.019796 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-b83b31b4-7f694d65c8-267bk" podUID="fdf63569-4cbd-40a9-a1e1-0dbf00641da6" containerName="payload-processing" containerID="cri-o://238b49261ffdce0b51bcc675b1e826c36219c43b272e07bee62432d549cc7508" gracePeriod=30 Jul 30 19:58:32.023488 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:32.023462 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b83b31b4-7857766488-8949s"] Jul 30 19:58:32.024238 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:32.024211 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-b83b31b4-7857766488-8949s" podUID="376f0e16-fa4e-492d-8a4a-9be579d96f65" containerName="payload-pre-processing" containerID="cri-o://5dd9db673480f638c5a47151f475dc4b2e951628e7cd5b22ca29a6ad7bcf2f42" gracePeriod=30 Jul 30 19:58:32.318057 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:32.318026 2556 generic.go:356] "Generic (PLEG): container finished" podID="fdf63569-4cbd-40a9-a1e1-0dbf00641da6" containerID="238b49261ffdce0b51bcc675b1e826c36219c43b272e07bee62432d549cc7508" exitCode=0 Jul 30 19:58:32.318526 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:32.318509 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-b83b31b4-7f694d65c8-267bk" Jul 30 19:58:32.319569 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:32.319544 2556 generic.go:356] "Generic (PLEG): container finished" podID="376f0e16-fa4e-492d-8a4a-9be579d96f65" containerID="5dd9db673480f638c5a47151f475dc4b2e951628e7cd5b22ca29a6ad7bcf2f42" exitCode=0 Jul 30 19:58:32.326140 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:32.326116 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-b83b31b4-7f694d65c8-267bk" event={"ID":"fdf63569-4cbd-40a9-a1e1-0dbf00641da6","Type":"ContainerDied","Data":"238b49261ffdce0b51bcc675b1e826c36219c43b272e07bee62432d549cc7508"} Jul 30 19:58:32.326225 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:32.326169 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-b83b31b4-7f694d65c8-267bk" event={"ID":"fdf63569-4cbd-40a9-a1e1-0dbf00641da6","Type":"ContainerDied","Data":"de163d6c03e04009774253dd795978c34bc1c495955f5c5657987d2a60c9da2f"} Jul 30 19:58:32.326225 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:32.326185 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-b83b31b4-7857766488-8949s" event={"ID":"376f0e16-fa4e-492d-8a4a-9be579d96f65","Type":"ContainerDied","Data":"5dd9db673480f638c5a47151f475dc4b2e951628e7cd5b22ca29a6ad7bcf2f42"} Jul 30 19:58:32.326225 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:32.326203 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-b83b31b4-7857766488-8949s" event={"ID":"376f0e16-fa4e-492d-8a4a-9be579d96f65","Type":"ContainerDied","Data":"e7e2fd9be6ba5c6be14df61d800f352cfcbcca2289b965009788fc2e5b5bcaeb"} Jul 30 19:58:32.326225 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:32.326217 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7e2fd9be6ba5c6be14df61d800f352cfcbcca2289b965009788fc2e5b5bcaeb" Jul 30 19:58:32.326420 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:32.326230 2556 scope.go:117] "RemoveContainer" containerID="238b49261ffdce0b51bcc675b1e826c36219c43b272e07bee62432d549cc7508" Jul 30 19:58:32.327470 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:32.327454 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-b83b31b4-7857766488-8949s" Jul 30 19:58:32.334638 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:32.334622 2556 scope.go:117] "RemoveContainer" containerID="238b49261ffdce0b51bcc675b1e826c36219c43b272e07bee62432d549cc7508" Jul 30 19:58:32.334878 ip-10-0-129-105 kubenswrapper[2556]: E0730 19:58:32.334859 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"238b49261ffdce0b51bcc675b1e826c36219c43b272e07bee62432d549cc7508\": container with ID starting with 238b49261ffdce0b51bcc675b1e826c36219c43b272e07bee62432d549cc7508 not found: ID does not exist" containerID="238b49261ffdce0b51bcc675b1e826c36219c43b272e07bee62432d549cc7508" Jul 30 19:58:32.334938 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:32.334891 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"238b49261ffdce0b51bcc675b1e826c36219c43b272e07bee62432d549cc7508"} err="failed to get container status \"238b49261ffdce0b51bcc675b1e826c36219c43b272e07bee62432d549cc7508\": rpc error: code = NotFound desc = could not find container \"238b49261ffdce0b51bcc675b1e826c36219c43b272e07bee62432d549cc7508\": container with ID starting with 238b49261ffdce0b51bcc675b1e826c36219c43b272e07bee62432d549cc7508 not found: ID does not exist" Jul 30 19:58:32.370159 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:32.370136 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hs8rx\" (UniqueName: \"kubernetes.io/projected/fdf63569-4cbd-40a9-a1e1-0dbf00641da6-kube-api-access-hs8rx\") pod \"fdf63569-4cbd-40a9-a1e1-0dbf00641da6\" (UID: \"fdf63569-4cbd-40a9-a1e1-0dbf00641da6\") " Jul 30 19:58:32.370261 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:32.370191 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/376f0e16-fa4e-492d-8a4a-9be579d96f65-plugins-config-volume\") pod \"376f0e16-fa4e-492d-8a4a-9be579d96f65\" (UID: \"376f0e16-fa4e-492d-8a4a-9be579d96f65\") " Jul 30 19:58:32.370261 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:32.370232 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdf63569-4cbd-40a9-a1e1-0dbf00641da6-plugins-config-volume\") pod \"fdf63569-4cbd-40a9-a1e1-0dbf00641da6\" (UID: \"fdf63569-4cbd-40a9-a1e1-0dbf00641da6\") " Jul 30 19:58:32.370372 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:32.370282 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5th2w\" (UniqueName: \"kubernetes.io/projected/376f0e16-fa4e-492d-8a4a-9be579d96f65-kube-api-access-5th2w\") pod \"376f0e16-fa4e-492d-8a4a-9be579d96f65\" (UID: \"376f0e16-fa4e-492d-8a4a-9be579d96f65\") " Jul 30 19:58:32.370624 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:32.370597 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/376f0e16-fa4e-492d-8a4a-9be579d96f65-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "376f0e16-fa4e-492d-8a4a-9be579d96f65" (UID: "376f0e16-fa4e-492d-8a4a-9be579d96f65"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 19:58:32.370999 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:32.370972 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdf63569-4cbd-40a9-a1e1-0dbf00641da6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fdf63569-4cbd-40a9-a1e1-0dbf00641da6" (UID: "fdf63569-4cbd-40a9-a1e1-0dbf00641da6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 19:58:32.372918 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:32.372898 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/376f0e16-fa4e-492d-8a4a-9be579d96f65-kube-api-access-5th2w" (OuterVolumeSpecName: "kube-api-access-5th2w") pod "376f0e16-fa4e-492d-8a4a-9be579d96f65" (UID: "376f0e16-fa4e-492d-8a4a-9be579d96f65"). InnerVolumeSpecName "kube-api-access-5th2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 19:58:32.372918 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:32.372904 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdf63569-4cbd-40a9-a1e1-0dbf00641da6-kube-api-access-hs8rx" (OuterVolumeSpecName: "kube-api-access-hs8rx") pod "fdf63569-4cbd-40a9-a1e1-0dbf00641da6" (UID: "fdf63569-4cbd-40a9-a1e1-0dbf00641da6"). InnerVolumeSpecName "kube-api-access-hs8rx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 19:58:32.471492 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:32.471458 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs8rx\" (UniqueName: \"kubernetes.io/projected/fdf63569-4cbd-40a9-a1e1-0dbf00641da6-kube-api-access-hs8rx\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:58:32.471492 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:32.471491 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/376f0e16-fa4e-492d-8a4a-9be579d96f65-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:58:32.471662 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:32.471508 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdf63569-4cbd-40a9-a1e1-0dbf00641da6-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:58:32.471662 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:32.471523 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5th2w\" (UniqueName: \"kubernetes.io/projected/376f0e16-fa4e-492d-8a4a-9be579d96f65-kube-api-access-5th2w\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 19:58:33.332865 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:33.332789 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-b83b31b4-7f694d65c8-267bk" Jul 30 19:58:33.333325 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:33.332796 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-b83b31b4-7857766488-8949s" Jul 30 19:58:33.355884 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:33.355856 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b83b31b4-7f694d65c8-267bk"] Jul 30 19:58:33.359239 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:33.359220 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b83b31b4-7f694d65c8-267bk"] Jul 30 19:58:33.371180 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:33.371157 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b83b31b4-7857766488-8949s"] Jul 30 19:58:33.374782 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:33.374762 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b83b31b4-7857766488-8949s"] Jul 30 19:58:34.324453 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:34.324423 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="376f0e16-fa4e-492d-8a4a-9be579d96f65" path="/var/lib/kubelet/pods/376f0e16-fa4e-492d-8a4a-9be579d96f65/volumes" Jul 30 19:58:34.324788 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:58:34.324776 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdf63569-4cbd-40a9-a1e1-0dbf00641da6" path="/var/lib/kubelet/pods/fdf63569-4cbd-40a9-a1e1-0dbf00641da6/volumes" Jul 30 19:59:38.372063 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:59:38.372029 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-795qw_4cdd383f-6493-4d83-80fa-f4ce17e8416e/ovn-acl-logging/0.log" Jul 30 19:59:38.375768 ip-10-0-129-105 kubenswrapper[2556]: I0730 19:59:38.375742 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-795qw_4cdd383f-6493-4d83-80fa-f4ce17e8416e/ovn-acl-logging/0.log" Jul 30 20:00:46.447156 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:00:46.447083 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-361f6a62-5849c59d88-gn42d"] Jul 30 20:00:46.447583 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:00:46.447509 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdf63569-4cbd-40a9-a1e1-0dbf00641da6" containerName="payload-processing" Jul 30 20:00:46.447583 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:00:46.447523 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdf63569-4cbd-40a9-a1e1-0dbf00641da6" containerName="payload-processing" Jul 30 20:00:46.447583 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:00:46.447532 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="376f0e16-fa4e-492d-8a4a-9be579d96f65" containerName="payload-pre-processing" Jul 30 20:00:46.447583 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:00:46.447539 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="376f0e16-fa4e-492d-8a4a-9be579d96f65" containerName="payload-pre-processing" Jul 30 20:00:46.447715 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:00:46.447602 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="376f0e16-fa4e-492d-8a4a-9be579d96f65" containerName="payload-pre-processing" Jul 30 20:00:46.447715 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:00:46.447610 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdf63569-4cbd-40a9-a1e1-0dbf00641da6" containerName="payload-processing" Jul 30 20:00:46.450192 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:00:46.450176 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-361f6a62-5849c59d88-gn42d" Jul 30 20:00:46.452639 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:00:46.452613 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-361f6a62-dockercfg-nsn2d\"" Jul 30 20:00:46.452982 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:00:46.452967 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-361f6a62\"" Jul 30 20:00:46.462620 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:00:46.462598 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-361f6a62-5849c59d88-gn42d"] Jul 30 20:00:46.602550 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:00:46.602515 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e3be97d-a856-4ecb-8dd8-20a45bb5ea22-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-361f6a62-5849c59d88-gn42d\" (UID: \"6e3be97d-a856-4ecb-8dd8-20a45bb5ea22\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-361f6a62-5849c59d88-gn42d" Jul 30 20:00:46.602709 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:00:46.602593 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkfc7\" (UniqueName: \"kubernetes.io/projected/6e3be97d-a856-4ecb-8dd8-20a45bb5ea22-kube-api-access-hkfc7\") pod \"payload-pre-processing-e2e-mt-361f6a62-5849c59d88-gn42d\" (UID: \"6e3be97d-a856-4ecb-8dd8-20a45bb5ea22\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-361f6a62-5849c59d88-gn42d" Jul 30 20:00:46.704013 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:00:46.703941 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e3be97d-a856-4ecb-8dd8-20a45bb5ea22-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-361f6a62-5849c59d88-gn42d\" (UID: \"6e3be97d-a856-4ecb-8dd8-20a45bb5ea22\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-361f6a62-5849c59d88-gn42d" Jul 30 20:00:46.704013 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:00:46.704004 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkfc7\" (UniqueName: \"kubernetes.io/projected/6e3be97d-a856-4ecb-8dd8-20a45bb5ea22-kube-api-access-hkfc7\") pod \"payload-pre-processing-e2e-mt-361f6a62-5849c59d88-gn42d\" (UID: \"6e3be97d-a856-4ecb-8dd8-20a45bb5ea22\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-361f6a62-5849c59d88-gn42d" Jul 30 20:00:46.704669 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:00:46.704652 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e3be97d-a856-4ecb-8dd8-20a45bb5ea22-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-361f6a62-5849c59d88-gn42d\" (UID: \"6e3be97d-a856-4ecb-8dd8-20a45bb5ea22\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-361f6a62-5849c59d88-gn42d" Jul 30 20:00:46.713757 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:00:46.713728 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkfc7\" (UniqueName: \"kubernetes.io/projected/6e3be97d-a856-4ecb-8dd8-20a45bb5ea22-kube-api-access-hkfc7\") pod \"payload-pre-processing-e2e-mt-361f6a62-5849c59d88-gn42d\" (UID: \"6e3be97d-a856-4ecb-8dd8-20a45bb5ea22\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-361f6a62-5849c59d88-gn42d" Jul 30 20:00:46.760587 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:00:46.760566 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-361f6a62-5849c59d88-gn42d" Jul 30 20:00:47.103607 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:00:47.103582 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-361f6a62-5849c59d88-gn42d"] Jul 30 20:00:47.105133 ip-10-0-129-105 kubenswrapper[2556]: W0730 20:00:47.105107 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e3be97d_a856_4ecb_8dd8_20a45bb5ea22.slice/crio-9035efab3ea7fafe366389e9353d42f1b31ea5bed74baa463056cb99f91eb9cc WatchSource:0}: Error finding container 9035efab3ea7fafe366389e9353d42f1b31ea5bed74baa463056cb99f91eb9cc: Status 404 returned error can't find the container with id 9035efab3ea7fafe366389e9353d42f1b31ea5bed74baa463056cb99f91eb9cc Jul 30 20:00:47.995174 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:00:47.995135 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-361f6a62-5849c59d88-gn42d" event={"ID":"6e3be97d-a856-4ecb-8dd8-20a45bb5ea22","Type":"ContainerStarted","Data":"5c233ea1137b05d4488bcd31990bc679b9cf95397a89bc6670e8b2aa14f08fc7"} Jul 30 20:00:47.995603 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:00:47.995197 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-361f6a62-5849c59d88-gn42d" event={"ID":"6e3be97d-a856-4ecb-8dd8-20a45bb5ea22","Type":"ContainerStarted","Data":"9035efab3ea7fafe366389e9353d42f1b31ea5bed74baa463056cb99f91eb9cc"} Jul 30 20:00:47.995603 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:00:47.995228 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-361f6a62-5849c59d88-gn42d" Jul 30 20:00:48.012712 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:00:48.012666 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-361f6a62-5849c59d88-gn42d" podStartSLOduration=2.012653099 podStartE2EDuration="2.012653099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:00:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:00:48.010277758 +0000 UTC m=+2170.229836672" watchObservedRunningTime="2026-07-30 20:00:48.012653099 +0000 UTC m=+2170.232212010" Jul 30 20:00:59.001884 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:00:59.001855 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-361f6a62-5849c59d88-gn42d" Jul 30 20:01:16.817767 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:01:16.817728 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-361f6a62-v4v68"] Jul 30 20:01:16.821337 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:01:16.821319 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-361f6a62-v4v68" Jul 30 20:01:16.824034 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:01:16.823947 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 30 20:01:16.824034 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:01:16.824008 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7t7kk\"" Jul 30 20:01:16.834626 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:01:16.834592 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-361f6a62-v4v68"] Jul 30 20:01:16.858819 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:01:16.858785 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc16d10f-339d-4107-902d-bda4d48e02b5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-361f6a62-v4v68\" (UID: \"cc16d10f-339d-4107-902d-bda4d48e02b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-361f6a62-v4v68" Jul 30 20:01:16.959351 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:01:16.959314 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc16d10f-339d-4107-902d-bda4d48e02b5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-361f6a62-v4v68\" (UID: \"cc16d10f-339d-4107-902d-bda4d48e02b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-361f6a62-v4v68" Jul 30 20:01:16.959953 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:01:16.959934 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc16d10f-339d-4107-902d-bda4d48e02b5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-361f6a62-v4v68\" (UID: \"cc16d10f-339d-4107-902d-bda4d48e02b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-361f6a62-v4v68" Jul 30 20:01:17.141139 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:01:17.141110 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-361f6a62-v4v68" Jul 30 20:01:17.278792 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:01:17.278765 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-361f6a62-v4v68"] Jul 30 20:01:17.280771 ip-10-0-129-105 kubenswrapper[2556]: W0730 20:01:17.280735 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc16d10f_339d_4107_902d_bda4d48e02b5.slice/crio-66717dccbd6205833939cf0f56c4e0bf712ef4829a3aae3aa942fcd7c2ae7aa0 WatchSource:0}: Error finding container 66717dccbd6205833939cf0f56c4e0bf712ef4829a3aae3aa942fcd7c2ae7aa0: Status 404 returned error can't find the container with id 66717dccbd6205833939cf0f56c4e0bf712ef4829a3aae3aa942fcd7c2ae7aa0 Jul 30 20:01:17.282938 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:01:17.282919 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 30 20:01:18.147543 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:01:18.147510 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-361f6a62-v4v68" event={"ID":"cc16d10f-339d-4107-902d-bda4d48e02b5","Type":"ContainerStarted","Data":"66717dccbd6205833939cf0f56c4e0bf712ef4829a3aae3aa942fcd7c2ae7aa0"} Jul 30 20:01:20.160461 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:01:20.160432 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-361f6a62-v4v68" event={"ID":"cc16d10f-339d-4107-902d-bda4d48e02b5","Type":"ContainerStarted","Data":"9afcc40351e7558ca82c0fe1b778b9866995176b0f5adb4026ce6538018cb65b"} Jul 30 20:01:20.179187 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:01:20.179148 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-361f6a62-v4v68" podStartSLOduration=1.7265588090000001 podStartE2EDuration="4.179135304s" totalImagesPullingTime="2.452576495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:01:20.176590123 +0000 UTC m=+2202.396149037" watchObservedRunningTime="2026-07-30 20:01:20.179135304 +0000 UTC m=+2202.398694216" Jul 30 20:01:21.167061 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:01:21.167029 2556 generic.go:356] "Generic (PLEG): container finished" podID="cc16d10f-339d-4107-902d-bda4d48e02b5" containerID="9afcc40351e7558ca82c0fe1b778b9866995176b0f5adb4026ce6538018cb65b" exitCode=0 Jul 30 20:01:21.167496 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:01:21.167115 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-361f6a62-v4v68" event={"ID":"cc16d10f-339d-4107-902d-bda4d48e02b5","Type":"ContainerDied","Data":"9afcc40351e7558ca82c0fe1b778b9866995176b0f5adb4026ce6538018cb65b"} Jul 30 20:01:22.306771 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:01:22.306744 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-361f6a62-v4v68" Jul 30 20:01:22.407804 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:01:22.407773 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc16d10f-339d-4107-902d-bda4d48e02b5-maas-api-service-ca\") pod \"cc16d10f-339d-4107-902d-bda4d48e02b5\" (UID: \"cc16d10f-339d-4107-902d-bda4d48e02b5\") " Jul 30 20:01:22.411487 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:01:22.408473 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc16d10f-339d-4107-902d-bda4d48e02b5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cc16d10f-339d-4107-902d-bda4d48e02b5" (UID: "cc16d10f-339d-4107-902d-bda4d48e02b5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 20:01:22.508762 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:01:22.508694 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc16d10f-339d-4107-902d-bda4d48e02b5-maas-api-service-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 20:01:23.179087 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:01:23.179060 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-361f6a62-v4v68" Jul 30 20:01:23.179266 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:01:23.179049 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-361f6a62-v4v68" event={"ID":"cc16d10f-339d-4107-902d-bda4d48e02b5","Type":"ContainerDied","Data":"66717dccbd6205833939cf0f56c4e0bf712ef4829a3aae3aa942fcd7c2ae7aa0"} Jul 30 20:01:23.179266 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:01:23.179214 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66717dccbd6205833939cf0f56c4e0bf712ef4829a3aae3aa942fcd7c2ae7aa0" Jul 30 20:01:27.271623 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:01:27.271587 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-361f6a62-5849c59d88-gn42d"] Jul 30 20:01:27.272093 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:01:27.271838 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-361f6a62-5849c59d88-gn42d" podUID="6e3be97d-a856-4ecb-8dd8-20a45bb5ea22" containerName="payload-pre-processing" containerID="cri-o://5c233ea1137b05d4488bcd31990bc679b9cf95397a89bc6670e8b2aa14f08fc7" gracePeriod=30 Jul 30 20:01:27.409941 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:01:27.409907 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-361f6a62-v4v68"] Jul 30 20:01:27.415827 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:01:27.415796 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-361f6a62-v4v68"] Jul 30 20:01:27.530876 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:01:27.530825 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-361f6a62-5849c59d88-gn42d" Jul 30 20:01:27.656155 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:01:27.656124 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hkfc7\" (UniqueName: \"kubernetes.io/projected/6e3be97d-a856-4ecb-8dd8-20a45bb5ea22-kube-api-access-hkfc7\") pod \"6e3be97d-a856-4ecb-8dd8-20a45bb5ea22\" (UID: \"6e3be97d-a856-4ecb-8dd8-20a45bb5ea22\") " Jul 30 20:01:27.656329 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:01:27.656167 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e3be97d-a856-4ecb-8dd8-20a45bb5ea22-plugins-config-volume\") pod \"6e3be97d-a856-4ecb-8dd8-20a45bb5ea22\" (UID: \"6e3be97d-a856-4ecb-8dd8-20a45bb5ea22\") " Jul 30 20:01:27.656572 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:01:27.656550 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e3be97d-a856-4ecb-8dd8-20a45bb5ea22-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6e3be97d-a856-4ecb-8dd8-20a45bb5ea22" (UID: "6e3be97d-a856-4ecb-8dd8-20a45bb5ea22"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 20:01:27.658179 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:01:27.658152 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e3be97d-a856-4ecb-8dd8-20a45bb5ea22-kube-api-access-hkfc7" (OuterVolumeSpecName: "kube-api-access-hkfc7") pod "6e3be97d-a856-4ecb-8dd8-20a45bb5ea22" (UID: "6e3be97d-a856-4ecb-8dd8-20a45bb5ea22"). InnerVolumeSpecName "kube-api-access-hkfc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:01:27.757281 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:01:27.757256 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkfc7\" (UniqueName: \"kubernetes.io/projected/6e3be97d-a856-4ecb-8dd8-20a45bb5ea22-kube-api-access-hkfc7\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 20:01:27.757281 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:01:27.757279 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e3be97d-a856-4ecb-8dd8-20a45bb5ea22-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 20:01:28.206102 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:01:28.206067 2556 generic.go:356] "Generic (PLEG): container finished" podID="6e3be97d-a856-4ecb-8dd8-20a45bb5ea22" containerID="5c233ea1137b05d4488bcd31990bc679b9cf95397a89bc6670e8b2aa14f08fc7" exitCode=0 Jul 30 20:01:28.206297 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:01:28.206123 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-361f6a62-5849c59d88-gn42d" Jul 30 20:01:28.206297 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:01:28.206150 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-361f6a62-5849c59d88-gn42d" event={"ID":"6e3be97d-a856-4ecb-8dd8-20a45bb5ea22","Type":"ContainerDied","Data":"5c233ea1137b05d4488bcd31990bc679b9cf95397a89bc6670e8b2aa14f08fc7"} Jul 30 20:01:28.206297 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:01:28.206205 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-361f6a62-5849c59d88-gn42d" event={"ID":"6e3be97d-a856-4ecb-8dd8-20a45bb5ea22","Type":"ContainerDied","Data":"9035efab3ea7fafe366389e9353d42f1b31ea5bed74baa463056cb99f91eb9cc"} Jul 30 20:01:28.206297 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:01:28.206229 2556 scope.go:117] "RemoveContainer" containerID="5c233ea1137b05d4488bcd31990bc679b9cf95397a89bc6670e8b2aa14f08fc7" Jul 30 20:01:28.217194 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:01:28.217172 2556 scope.go:117] "RemoveContainer" containerID="5c233ea1137b05d4488bcd31990bc679b9cf95397a89bc6670e8b2aa14f08fc7" Jul 30 20:01:28.217543 ip-10-0-129-105 kubenswrapper[2556]: E0730 20:01:28.217524 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c233ea1137b05d4488bcd31990bc679b9cf95397a89bc6670e8b2aa14f08fc7\": container with ID starting with 5c233ea1137b05d4488bcd31990bc679b9cf95397a89bc6670e8b2aa14f08fc7 not found: ID does not exist" containerID="5c233ea1137b05d4488bcd31990bc679b9cf95397a89bc6670e8b2aa14f08fc7" Jul 30 20:01:28.217638 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:01:28.217554 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c233ea1137b05d4488bcd31990bc679b9cf95397a89bc6670e8b2aa14f08fc7"} err="failed to get container status \"5c233ea1137b05d4488bcd31990bc679b9cf95397a89bc6670e8b2aa14f08fc7\": rpc error: code = NotFound desc = could not find container \"5c233ea1137b05d4488bcd31990bc679b9cf95397a89bc6670e8b2aa14f08fc7\": container with ID starting with 5c233ea1137b05d4488bcd31990bc679b9cf95397a89bc6670e8b2aa14f08fc7 not found: ID does not exist" Jul 30 20:01:28.229766 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:01:28.229743 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-361f6a62-5849c59d88-gn42d"] Jul 30 20:01:28.233511 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:01:28.233490 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-361f6a62-5849c59d88-gn42d"] Jul 30 20:01:28.323790 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:01:28.323768 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e3be97d-a856-4ecb-8dd8-20a45bb5ea22" path="/var/lib/kubelet/pods/6e3be97d-a856-4ecb-8dd8-20a45bb5ea22/volumes" Jul 30 20:01:28.324096 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:01:28.324062 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc16d10f-339d-4107-902d-bda4d48e02b5" path="/var/lib/kubelet/pods/cc16d10f-339d-4107-902d-bda4d48e02b5/volumes" Jul 30 20:03:53.090572 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:03:53.090535 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f6ff274e-698b744b7b-7h4p4"] Jul 30 20:03:53.091042 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:03:53.091024 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e3be97d-a856-4ecb-8dd8-20a45bb5ea22" containerName="payload-pre-processing" Jul 30 20:03:53.091091 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:03:53.091044 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e3be97d-a856-4ecb-8dd8-20a45bb5ea22" containerName="payload-pre-processing" Jul 30 20:03:53.091091 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:03:53.091059 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc16d10f-339d-4107-902d-bda4d48e02b5" containerName="revoke-keys" Jul 30 20:03:53.091091 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:03:53.091065 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc16d10f-339d-4107-902d-bda4d48e02b5" containerName="revoke-keys" Jul 30 20:03:53.091188 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:03:53.091147 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e3be97d-a856-4ecb-8dd8-20a45bb5ea22" containerName="payload-pre-processing" Jul 30 20:03:53.091188 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:03:53.091159 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc16d10f-339d-4107-902d-bda4d48e02b5" containerName="revoke-keys" Jul 30 20:03:53.094346 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:03:53.094328 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f6ff274e-698b744b7b-7h4p4" Jul 30 20:03:53.097229 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:03:53.097205 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-f6ff274e-dockercfg-k7ft9\"" Jul 30 20:03:53.097631 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:03:53.097614 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-f6ff274e\"" Jul 30 20:03:53.110547 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:03:53.110522 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f6ff274e-698b744b7b-7h4p4"] Jul 30 20:03:53.213727 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:03:53.213693 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70f0f975-7328-4e49-a5f5-8f6ea553c035-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f6ff274e-698b744b7b-7h4p4\" (UID: \"70f0f975-7328-4e49-a5f5-8f6ea553c035\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f6ff274e-698b744b7b-7h4p4" Jul 30 20:03:53.213906 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:03:53.213757 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rkgr\" (UniqueName: \"kubernetes.io/projected/70f0f975-7328-4e49-a5f5-8f6ea553c035-kube-api-access-9rkgr\") pod \"payload-pre-processing-e2e-mt-f6ff274e-698b744b7b-7h4p4\" (UID: \"70f0f975-7328-4e49-a5f5-8f6ea553c035\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f6ff274e-698b744b7b-7h4p4" Jul 30 20:03:53.314772 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:03:53.314729 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70f0f975-7328-4e49-a5f5-8f6ea553c035-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f6ff274e-698b744b7b-7h4p4\" (UID: \"70f0f975-7328-4e49-a5f5-8f6ea553c035\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f6ff274e-698b744b7b-7h4p4" Jul 30 20:03:53.314940 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:03:53.314800 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rkgr\" (UniqueName: \"kubernetes.io/projected/70f0f975-7328-4e49-a5f5-8f6ea553c035-kube-api-access-9rkgr\") pod \"payload-pre-processing-e2e-mt-f6ff274e-698b744b7b-7h4p4\" (UID: \"70f0f975-7328-4e49-a5f5-8f6ea553c035\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f6ff274e-698b744b7b-7h4p4" Jul 30 20:03:53.315427 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:03:53.315410 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70f0f975-7328-4e49-a5f5-8f6ea553c035-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f6ff274e-698b744b7b-7h4p4\" (UID: \"70f0f975-7328-4e49-a5f5-8f6ea553c035\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f6ff274e-698b744b7b-7h4p4" Jul 30 20:03:53.322607 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:03:53.322581 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rkgr\" (UniqueName: \"kubernetes.io/projected/70f0f975-7328-4e49-a5f5-8f6ea553c035-kube-api-access-9rkgr\") pod \"payload-pre-processing-e2e-mt-f6ff274e-698b744b7b-7h4p4\" (UID: \"70f0f975-7328-4e49-a5f5-8f6ea553c035\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f6ff274e-698b744b7b-7h4p4" Jul 30 20:03:53.405370 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:03:53.405338 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f6ff274e-698b744b7b-7h4p4" Jul 30 20:03:53.749654 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:03:53.749629 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f6ff274e-698b744b7b-7h4p4"] Jul 30 20:03:53.751942 ip-10-0-129-105 kubenswrapper[2556]: W0730 20:03:53.751911 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70f0f975_7328_4e49_a5f5_8f6ea553c035.slice/crio-4b88a5179367948babdc28c7e0459800d42d4cac984d8f1b81ad17084d081cb1 WatchSource:0}: Error finding container 4b88a5179367948babdc28c7e0459800d42d4cac984d8f1b81ad17084d081cb1: Status 404 returned error can't find the container with id 4b88a5179367948babdc28c7e0459800d42d4cac984d8f1b81ad17084d081cb1 Jul 30 20:03:53.936071 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:03:53.936038 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f6ff274e-698b744b7b-7h4p4" event={"ID":"70f0f975-7328-4e49-a5f5-8f6ea553c035","Type":"ContainerStarted","Data":"0ad311b5accc5a2c620060140f00a00c8f37521d5a6ce461cff0c7fff57fa5c7"} Jul 30 20:03:53.936253 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:03:53.936096 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f6ff274e-698b744b7b-7h4p4" event={"ID":"70f0f975-7328-4e49-a5f5-8f6ea553c035","Type":"ContainerStarted","Data":"4b88a5179367948babdc28c7e0459800d42d4cac984d8f1b81ad17084d081cb1"} Jul 30 20:03:53.936253 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:03:53.936144 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f6ff274e-698b744b7b-7h4p4" Jul 30 20:03:53.955491 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:03:53.955443 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-f6ff274e-698b744b7b-7h4p4" podStartSLOduration=0.95542836 podStartE2EDuration="955.42836ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:03:53.953612105 +0000 UTC m=+2356.173171019" watchObservedRunningTime="2026-07-30 20:03:53.95542836 +0000 UTC m=+2356.174987273" Jul 30 20:04:04.942473 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:04:04.942430 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-f6ff274e-698b744b7b-7h4p4" podUID="70f0f975-7328-4e49-a5f5-8f6ea553c035" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.51:9004: connect: connection refused" Jul 30 20:04:14.942408 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:04:14.942350 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f6ff274e-698b744b7b-7h4p4" Jul 30 20:04:38.406795 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:04:38.406771 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-795qw_4cdd383f-6493-4d83-80fa-f4ce17e8416e/ovn-acl-logging/0.log" Jul 30 20:04:38.411328 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:04:38.411302 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-795qw_4cdd383f-6493-4d83-80fa-f4ce17e8416e/ovn-acl-logging/0.log" Jul 30 20:04:38.493987 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:04:38.493958 2556 scope.go:117] "RemoveContainer" containerID="5dd9db673480f638c5a47151f475dc4b2e951628e7cd5b22ca29a6ad7bcf2f42" Jul 30 20:04:55.959945 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:04:55.959913 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f6ff274e-fhw27"] Jul 30 20:04:55.963458 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:04:55.963438 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f6ff274e-fhw27" Jul 30 20:04:55.965849 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:04:55.965824 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7t7kk\"" Jul 30 20:04:55.965966 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:04:55.965824 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 30 20:04:55.970240 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:04:55.970212 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f6ff274e-fhw27"] Jul 30 20:04:56.092366 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:04:56.092337 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a360e260-ee82-4c9e-820c-7275bfed2473-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f6ff274e-fhw27\" (UID: \"a360e260-ee82-4c9e-820c-7275bfed2473\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f6ff274e-fhw27" Jul 30 20:04:56.193404 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:04:56.193354 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a360e260-ee82-4c9e-820c-7275bfed2473-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f6ff274e-fhw27\" (UID: \"a360e260-ee82-4c9e-820c-7275bfed2473\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f6ff274e-fhw27" Jul 30 20:04:56.193944 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:04:56.193927 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a360e260-ee82-4c9e-820c-7275bfed2473-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f6ff274e-fhw27\" (UID: \"a360e260-ee82-4c9e-820c-7275bfed2473\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f6ff274e-fhw27" Jul 30 20:04:56.276054 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:04:56.275996 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f6ff274e-fhw27" Jul 30 20:04:56.406278 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:04:56.406246 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f6ff274e-fhw27"] Jul 30 20:04:56.407438 ip-10-0-129-105 kubenswrapper[2556]: W0730 20:04:56.407409 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda360e260_ee82_4c9e_820c_7275bfed2473.slice/crio-3d1aed12c93f7fd371acfdb05e17446de093fce157100dd2e16827c73bba6b23 WatchSource:0}: Error finding container 3d1aed12c93f7fd371acfdb05e17446de093fce157100dd2e16827c73bba6b23: Status 404 returned error can't find the container with id 3d1aed12c93f7fd371acfdb05e17446de093fce157100dd2e16827c73bba6b23 Jul 30 20:04:57.259928 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:04:57.259895 2556 generic.go:356] "Generic (PLEG): container finished" podID="a360e260-ee82-4c9e-820c-7275bfed2473" containerID="43bf3b36cfb3b90136bc334d7778e5da8981fa72a00118587413e7d43c114295" exitCode=0 Jul 30 20:04:57.260279 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:04:57.259980 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f6ff274e-fhw27" event={"ID":"a360e260-ee82-4c9e-820c-7275bfed2473","Type":"ContainerDied","Data":"43bf3b36cfb3b90136bc334d7778e5da8981fa72a00118587413e7d43c114295"} Jul 30 20:04:57.260279 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:04:57.260055 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f6ff274e-fhw27" event={"ID":"a360e260-ee82-4c9e-820c-7275bfed2473","Type":"ContainerStarted","Data":"3d1aed12c93f7fd371acfdb05e17446de093fce157100dd2e16827c73bba6b23"} Jul 30 20:04:58.403679 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:04:58.403651 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f6ff274e-fhw27" Jul 30 20:04:58.513449 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:04:58.513421 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a360e260-ee82-4c9e-820c-7275bfed2473-maas-api-service-ca\") pod \"a360e260-ee82-4c9e-820c-7275bfed2473\" (UID: \"a360e260-ee82-4c9e-820c-7275bfed2473\") " Jul 30 20:04:58.513782 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:04:58.513751 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a360e260-ee82-4c9e-820c-7275bfed2473-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a360e260-ee82-4c9e-820c-7275bfed2473" (UID: "a360e260-ee82-4c9e-820c-7275bfed2473"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 20:04:58.614653 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:04:58.614593 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a360e260-ee82-4c9e-820c-7275bfed2473-maas-api-service-ca\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 20:04:59.272082 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:04:59.272054 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f6ff274e-fhw27" Jul 30 20:04:59.272262 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:04:59.272106 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f6ff274e-fhw27" event={"ID":"a360e260-ee82-4c9e-820c-7275bfed2473","Type":"ContainerDied","Data":"3d1aed12c93f7fd371acfdb05e17446de093fce157100dd2e16827c73bba6b23"} Jul 30 20:04:59.272262 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:04:59.272152 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d1aed12c93f7fd371acfdb05e17446de093fce157100dd2e16827c73bba6b23" Jul 30 20:05:11.464810 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:05:11.464723 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f6ff274e-698b744b7b-7h4p4"] Jul 30 20:05:11.465459 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:05:11.464968 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-f6ff274e-698b744b7b-7h4p4" podUID="70f0f975-7328-4e49-a5f5-8f6ea553c035" containerName="payload-pre-processing" containerID="cri-o://0ad311b5accc5a2c620060140f00a00c8f37521d5a6ce461cff0c7fff57fa5c7" gracePeriod=30 Jul 30 20:05:11.595732 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:05:11.595701 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f6ff274e-fhw27"] Jul 30 20:05:11.598367 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:05:11.598340 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f6ff274e-fhw27"] Jul 30 20:05:11.717979 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:05:11.717918 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f6ff274e-698b744b7b-7h4p4" Jul 30 20:05:11.828466 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:05:11.828440 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70f0f975-7328-4e49-a5f5-8f6ea553c035-plugins-config-volume\") pod \"70f0f975-7328-4e49-a5f5-8f6ea553c035\" (UID: \"70f0f975-7328-4e49-a5f5-8f6ea553c035\") " Jul 30 20:05:11.828626 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:05:11.828481 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rkgr\" (UniqueName: \"kubernetes.io/projected/70f0f975-7328-4e49-a5f5-8f6ea553c035-kube-api-access-9rkgr\") pod \"70f0f975-7328-4e49-a5f5-8f6ea553c035\" (UID: \"70f0f975-7328-4e49-a5f5-8f6ea553c035\") " Jul 30 20:05:11.828840 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:05:11.828816 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70f0f975-7328-4e49-a5f5-8f6ea553c035-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "70f0f975-7328-4e49-a5f5-8f6ea553c035" (UID: "70f0f975-7328-4e49-a5f5-8f6ea553c035"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 20:05:11.830499 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:05:11.830473 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70f0f975-7328-4e49-a5f5-8f6ea553c035-kube-api-access-9rkgr" (OuterVolumeSpecName: "kube-api-access-9rkgr") pod "70f0f975-7328-4e49-a5f5-8f6ea553c035" (UID: "70f0f975-7328-4e49-a5f5-8f6ea553c035"). InnerVolumeSpecName "kube-api-access-9rkgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:05:11.929093 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:05:11.929070 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rkgr\" (UniqueName: \"kubernetes.io/projected/70f0f975-7328-4e49-a5f5-8f6ea553c035-kube-api-access-9rkgr\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 20:05:11.929093 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:05:11.929093 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70f0f975-7328-4e49-a5f5-8f6ea553c035-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 20:05:12.324548 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:05:12.324515 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a360e260-ee82-4c9e-820c-7275bfed2473" path="/var/lib/kubelet/pods/a360e260-ee82-4c9e-820c-7275bfed2473/volumes" Jul 30 20:05:12.344684 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:05:12.344658 2556 generic.go:356] "Generic (PLEG): container finished" podID="70f0f975-7328-4e49-a5f5-8f6ea553c035" containerID="0ad311b5accc5a2c620060140f00a00c8f37521d5a6ce461cff0c7fff57fa5c7" exitCode=0 Jul 30 20:05:12.344792 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:05:12.344706 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f6ff274e-698b744b7b-7h4p4" Jul 30 20:05:12.344792 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:05:12.344737 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f6ff274e-698b744b7b-7h4p4" event={"ID":"70f0f975-7328-4e49-a5f5-8f6ea553c035","Type":"ContainerDied","Data":"0ad311b5accc5a2c620060140f00a00c8f37521d5a6ce461cff0c7fff57fa5c7"} Jul 30 20:05:12.344866 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:05:12.344792 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f6ff274e-698b744b7b-7h4p4" event={"ID":"70f0f975-7328-4e49-a5f5-8f6ea553c035","Type":"ContainerDied","Data":"4b88a5179367948babdc28c7e0459800d42d4cac984d8f1b81ad17084d081cb1"} Jul 30 20:05:12.344866 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:05:12.344823 2556 scope.go:117] "RemoveContainer" containerID="0ad311b5accc5a2c620060140f00a00c8f37521d5a6ce461cff0c7fff57fa5c7" Jul 30 20:05:12.353233 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:05:12.353214 2556 scope.go:117] "RemoveContainer" containerID="0ad311b5accc5a2c620060140f00a00c8f37521d5a6ce461cff0c7fff57fa5c7" Jul 30 20:05:12.353500 ip-10-0-129-105 kubenswrapper[2556]: E0730 20:05:12.353481 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ad311b5accc5a2c620060140f00a00c8f37521d5a6ce461cff0c7fff57fa5c7\": container with ID starting with 0ad311b5accc5a2c620060140f00a00c8f37521d5a6ce461cff0c7fff57fa5c7 not found: ID does not exist" containerID="0ad311b5accc5a2c620060140f00a00c8f37521d5a6ce461cff0c7fff57fa5c7" Jul 30 20:05:12.353550 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:05:12.353508 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ad311b5accc5a2c620060140f00a00c8f37521d5a6ce461cff0c7fff57fa5c7"} err="failed to get container status \"0ad311b5accc5a2c620060140f00a00c8f37521d5a6ce461cff0c7fff57fa5c7\": rpc error: code = NotFound desc = could not find container \"0ad311b5accc5a2c620060140f00a00c8f37521d5a6ce461cff0c7fff57fa5c7\": container with ID starting with 0ad311b5accc5a2c620060140f00a00c8f37521d5a6ce461cff0c7fff57fa5c7 not found: ID does not exist" Jul 30 20:05:12.360464 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:05:12.360442 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f6ff274e-698b744b7b-7h4p4"] Jul 30 20:05:12.362268 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:05:12.362250 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f6ff274e-698b744b7b-7h4p4"] Jul 30 20:05:14.324243 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:05:14.324211 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70f0f975-7328-4e49-a5f5-8f6ea553c035" path="/var/lib/kubelet/pods/70f0f975-7328-4e49-a5f5-8f6ea553c035/volumes" Jul 30 20:07:16.908540 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:16.908500 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-37abda-bb4d897bd-s5rrz"] Jul 30 20:07:16.909116 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:16.909096 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70f0f975-7328-4e49-a5f5-8f6ea553c035" containerName="payload-pre-processing" Jul 30 20:07:16.909206 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:16.909119 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="70f0f975-7328-4e49-a5f5-8f6ea553c035" containerName="payload-pre-processing" Jul 30 20:07:16.909206 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:16.909171 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a360e260-ee82-4c9e-820c-7275bfed2473" containerName="revoke-keys" Jul 30 20:07:16.909206 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:16.909180 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a360e260-ee82-4c9e-820c-7275bfed2473" containerName="revoke-keys" Jul 30 20:07:16.909364 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:16.909298 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="70f0f975-7328-4e49-a5f5-8f6ea553c035" containerName="payload-pre-processing" Jul 30 20:07:16.909364 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:16.909312 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a360e260-ee82-4c9e-820c-7275bfed2473" containerName="revoke-keys" Jul 30 20:07:16.912596 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:16.912576 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-37abda-bb4d897bd-s5rrz" Jul 30 20:07:16.915276 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:16.915253 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-37abda\"" Jul 30 20:07:16.915405 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:16.915350 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-37abda-dockercfg-8qkwp\"" Jul 30 20:07:16.922673 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:16.922649 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-37abda-bb4d897bd-s5rrz"] Jul 30 20:07:16.990829 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:16.990795 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/152306ac-f28a-44ea-964e-47a936514063-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-37abda-bb4d897bd-s5rrz\" (UID: \"152306ac-f28a-44ea-964e-47a936514063\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-37abda-bb4d897bd-s5rrz" Jul 30 20:07:16.990988 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:16.990855 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g55xq\" (UniqueName: \"kubernetes.io/projected/152306ac-f28a-44ea-964e-47a936514063-kube-api-access-g55xq\") pod \"payload-pre-processing-e2e-inf-a-37abda-bb4d897bd-s5rrz\" (UID: \"152306ac-f28a-44ea-964e-47a936514063\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-37abda-bb4d897bd-s5rrz" Jul 30 20:07:17.091625 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:17.091577 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/152306ac-f28a-44ea-964e-47a936514063-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-37abda-bb4d897bd-s5rrz\" (UID: \"152306ac-f28a-44ea-964e-47a936514063\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-37abda-bb4d897bd-s5rrz" Jul 30 20:07:17.091625 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:17.091636 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g55xq\" (UniqueName: \"kubernetes.io/projected/152306ac-f28a-44ea-964e-47a936514063-kube-api-access-g55xq\") pod \"payload-pre-processing-e2e-inf-a-37abda-bb4d897bd-s5rrz\" (UID: \"152306ac-f28a-44ea-964e-47a936514063\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-37abda-bb4d897bd-s5rrz" Jul 30 20:07:17.092331 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:17.092306 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/152306ac-f28a-44ea-964e-47a936514063-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-37abda-bb4d897bd-s5rrz\" (UID: \"152306ac-f28a-44ea-964e-47a936514063\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-37abda-bb4d897bd-s5rrz" Jul 30 20:07:17.100630 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:17.100608 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g55xq\" (UniqueName: \"kubernetes.io/projected/152306ac-f28a-44ea-964e-47a936514063-kube-api-access-g55xq\") pod \"payload-pre-processing-e2e-inf-a-37abda-bb4d897bd-s5rrz\" (UID: \"152306ac-f28a-44ea-964e-47a936514063\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-37abda-bb4d897bd-s5rrz" Jul 30 20:07:17.223717 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:17.223645 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-37abda-bb4d897bd-s5rrz" Jul 30 20:07:17.573473 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:17.573433 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-37abda-bb4d897bd-s5rrz"] Jul 30 20:07:17.574969 ip-10-0-129-105 kubenswrapper[2556]: W0730 20:07:17.574944 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod152306ac_f28a_44ea_964e_47a936514063.slice/crio-b5db1ffd755e717137ce992ac710b1af4d95cd5967ca5d2e529534167c76087b WatchSource:0}: Error finding container b5db1ffd755e717137ce992ac710b1af4d95cd5967ca5d2e529534167c76087b: Status 404 returned error can't find the container with id b5db1ffd755e717137ce992ac710b1af4d95cd5967ca5d2e529534167c76087b Jul 30 20:07:17.966931 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:17.966898 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-37abda-bb4d897bd-s5rrz" event={"ID":"152306ac-f28a-44ea-964e-47a936514063","Type":"ContainerStarted","Data":"0ae957de5348b176bb89aa7c63217b7b8c0a45053628ae2893bb7a09f7d49fa5"} Jul 30 20:07:17.967306 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:17.966945 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-37abda-bb4d897bd-s5rrz" event={"ID":"152306ac-f28a-44ea-964e-47a936514063","Type":"ContainerStarted","Data":"b5db1ffd755e717137ce992ac710b1af4d95cd5967ca5d2e529534167c76087b"} Jul 30 20:07:17.967306 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:17.967022 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-37abda-bb4d897bd-s5rrz" Jul 30 20:07:17.985805 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:17.985765 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-37abda-bb4d897bd-s5rrz" podStartSLOduration=1.985751619 podStartE2EDuration="1.985751619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:07:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:07:17.984505938 +0000 UTC m=+2560.204064854" watchObservedRunningTime="2026-07-30 20:07:17.985751619 +0000 UTC m=+2560.205310533" Jul 30 20:07:28.975552 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:28.975520 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-37abda-bb4d897bd-s5rrz" Jul 30 20:07:38.534258 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:38.534230 2556 scope.go:117] "RemoveContainer" containerID="9afcc40351e7558ca82c0fe1b778b9866995176b0f5adb4026ce6538018cb65b" Jul 30 20:07:46.126547 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:46.126504 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ff2f84-8d4bb8d4-7nd98"] Jul 30 20:07:46.129977 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:46.129962 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ff2f84-8d4bb8d4-7nd98" Jul 30 20:07:46.132451 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:46.132427 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-ff2f84-dockercfg-wqb6v\"" Jul 30 20:07:46.132588 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:46.132456 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-ff2f84\"" Jul 30 20:07:46.140019 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:46.139999 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ff2f84-8d4bb8d4-7nd98"] Jul 30 20:07:46.164246 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:46.164225 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ff2f84-96877d8b4-bpxv8"] Jul 30 20:07:46.167729 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:46.167713 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ff2f84-96877d8b4-bpxv8" Jul 30 20:07:46.175309 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:46.175285 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ff2f84-96877d8b4-bpxv8"] Jul 30 20:07:46.264555 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:46.264522 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spmv6\" (UniqueName: \"kubernetes.io/projected/f5fc335d-498f-4be3-825e-fff2d0ee3b3d-kube-api-access-spmv6\") pod \"payload-processing-e2e-inf-b-ff2f84-96877d8b4-bpxv8\" (UID: \"f5fc335d-498f-4be3-825e-fff2d0ee3b3d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ff2f84-96877d8b4-bpxv8" Jul 30 20:07:46.264710 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:46.264574 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84hk2\" (UniqueName: \"kubernetes.io/projected/35bbbd1c-5a81-4e0f-bd7d-a1b20e95e1d3-kube-api-access-84hk2\") pod \"payload-pre-processing-e2e-inf-b-ff2f84-8d4bb8d4-7nd98\" (UID: \"35bbbd1c-5a81-4e0f-bd7d-a1b20e95e1d3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ff2f84-8d4bb8d4-7nd98" Jul 30 20:07:46.264710 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:46.264640 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35bbbd1c-5a81-4e0f-bd7d-a1b20e95e1d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ff2f84-8d4bb8d4-7nd98\" (UID: \"35bbbd1c-5a81-4e0f-bd7d-a1b20e95e1d3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ff2f84-8d4bb8d4-7nd98" Jul 30 20:07:46.264855 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:46.264722 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5fc335d-498f-4be3-825e-fff2d0ee3b3d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ff2f84-96877d8b4-bpxv8\" (UID: \"f5fc335d-498f-4be3-825e-fff2d0ee3b3d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ff2f84-96877d8b4-bpxv8" Jul 30 20:07:46.365856 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:46.365829 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5fc335d-498f-4be3-825e-fff2d0ee3b3d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ff2f84-96877d8b4-bpxv8\" (UID: \"f5fc335d-498f-4be3-825e-fff2d0ee3b3d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ff2f84-96877d8b4-bpxv8" Jul 30 20:07:46.365989 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:46.365968 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spmv6\" (UniqueName: \"kubernetes.io/projected/f5fc335d-498f-4be3-825e-fff2d0ee3b3d-kube-api-access-spmv6\") pod \"payload-processing-e2e-inf-b-ff2f84-96877d8b4-bpxv8\" (UID: \"f5fc335d-498f-4be3-825e-fff2d0ee3b3d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ff2f84-96877d8b4-bpxv8" Jul 30 20:07:46.366028 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:46.366013 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84hk2\" (UniqueName: \"kubernetes.io/projected/35bbbd1c-5a81-4e0f-bd7d-a1b20e95e1d3-kube-api-access-84hk2\") pod \"payload-pre-processing-e2e-inf-b-ff2f84-8d4bb8d4-7nd98\" (UID: \"35bbbd1c-5a81-4e0f-bd7d-a1b20e95e1d3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ff2f84-8d4bb8d4-7nd98" Jul 30 20:07:46.366064 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:46.366054 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35bbbd1c-5a81-4e0f-bd7d-a1b20e95e1d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ff2f84-8d4bb8d4-7nd98\" (UID: \"35bbbd1c-5a81-4e0f-bd7d-a1b20e95e1d3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ff2f84-8d4bb8d4-7nd98" Jul 30 20:07:46.366797 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:46.366774 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5fc335d-498f-4be3-825e-fff2d0ee3b3d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ff2f84-96877d8b4-bpxv8\" (UID: \"f5fc335d-498f-4be3-825e-fff2d0ee3b3d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ff2f84-96877d8b4-bpxv8" Jul 30 20:07:46.366901 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:46.366823 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35bbbd1c-5a81-4e0f-bd7d-a1b20e95e1d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ff2f84-8d4bb8d4-7nd98\" (UID: \"35bbbd1c-5a81-4e0f-bd7d-a1b20e95e1d3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ff2f84-8d4bb8d4-7nd98" Jul 30 20:07:46.373851 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:46.373830 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spmv6\" (UniqueName: \"kubernetes.io/projected/f5fc335d-498f-4be3-825e-fff2d0ee3b3d-kube-api-access-spmv6\") pod \"payload-processing-e2e-inf-b-ff2f84-96877d8b4-bpxv8\" (UID: \"f5fc335d-498f-4be3-825e-fff2d0ee3b3d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ff2f84-96877d8b4-bpxv8" Jul 30 20:07:46.373981 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:46.373960 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84hk2\" (UniqueName: \"kubernetes.io/projected/35bbbd1c-5a81-4e0f-bd7d-a1b20e95e1d3-kube-api-access-84hk2\") pod \"payload-pre-processing-e2e-inf-b-ff2f84-8d4bb8d4-7nd98\" (UID: \"35bbbd1c-5a81-4e0f-bd7d-a1b20e95e1d3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ff2f84-8d4bb8d4-7nd98" Jul 30 20:07:46.441185 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:46.441129 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ff2f84-8d4bb8d4-7nd98" Jul 30 20:07:46.480213 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:46.480179 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ff2f84-96877d8b4-bpxv8" Jul 30 20:07:46.590217 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:46.590184 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ff2f84-8d4bb8d4-7nd98"] Jul 30 20:07:46.593480 ip-10-0-129-105 kubenswrapper[2556]: W0730 20:07:46.593448 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35bbbd1c_5a81_4e0f_bd7d_a1b20e95e1d3.slice/crio-e5ed2d936e03745c07a4e33830de5840afbbbcaa8d32bd55863221c05b772bfe WatchSource:0}: Error finding container e5ed2d936e03745c07a4e33830de5840afbbbcaa8d32bd55863221c05b772bfe: Status 404 returned error can't find the container with id e5ed2d936e03745c07a4e33830de5840afbbbcaa8d32bd55863221c05b772bfe Jul 30 20:07:46.642726 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:46.642704 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ff2f84-96877d8b4-bpxv8"] Jul 30 20:07:46.644421 ip-10-0-129-105 kubenswrapper[2556]: W0730 20:07:46.644397 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5fc335d_498f_4be3_825e_fff2d0ee3b3d.slice/crio-b5edeefcc4858b0cd616cfddaa8d0a4808604aa9720c7e971b0bb64dfb92b50f WatchSource:0}: Error finding container b5edeefcc4858b0cd616cfddaa8d0a4808604aa9720c7e971b0bb64dfb92b50f: Status 404 returned error can't find the container with id b5edeefcc4858b0cd616cfddaa8d0a4808604aa9720c7e971b0bb64dfb92b50f Jul 30 20:07:47.120102 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:47.119995 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ff2f84-96877d8b4-bpxv8" event={"ID":"f5fc335d-498f-4be3-825e-fff2d0ee3b3d","Type":"ContainerStarted","Data":"74f61c1a689cea8798c6447a61ebb5b6aaa493606cfa08358299615a8115e9c2"} Jul 30 20:07:47.120102 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:47.120061 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ff2f84-96877d8b4-bpxv8" event={"ID":"f5fc335d-498f-4be3-825e-fff2d0ee3b3d","Type":"ContainerStarted","Data":"b5edeefcc4858b0cd616cfddaa8d0a4808604aa9720c7e971b0bb64dfb92b50f"} Jul 30 20:07:47.120353 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:47.120259 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ff2f84-96877d8b4-bpxv8" Jul 30 20:07:47.121835 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:47.121808 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ff2f84-8d4bb8d4-7nd98" event={"ID":"35bbbd1c-5a81-4e0f-bd7d-a1b20e95e1d3","Type":"ContainerStarted","Data":"559acaf1f685ada1c484e8d7396443e67aa6c81ba294c3a58bed17c64d6185d9"} Jul 30 20:07:47.121973 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:47.121848 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ff2f84-8d4bb8d4-7nd98" event={"ID":"35bbbd1c-5a81-4e0f-bd7d-a1b20e95e1d3","Type":"ContainerStarted","Data":"e5ed2d936e03745c07a4e33830de5840afbbbcaa8d32bd55863221c05b772bfe"} Jul 30 20:07:47.121973 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:47.121949 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ff2f84-8d4bb8d4-7nd98" Jul 30 20:07:47.138547 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:47.138504 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-ff2f84-96877d8b4-bpxv8" podStartSLOduration=1.138491021 podStartE2EDuration="1.138491021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:07:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:07:47.136124038 +0000 UTC m=+2589.355682934" watchObservedRunningTime="2026-07-30 20:07:47.138491021 +0000 UTC m=+2589.358049943" Jul 30 20:07:47.153518 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:47.153470 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ff2f84-8d4bb8d4-7nd98" podStartSLOduration=1.1534543529999999 podStartE2EDuration="1.153454353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:07:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:07:47.150172487 +0000 UTC m=+2589.369731425" watchObservedRunningTime="2026-07-30 20:07:47.153454353 +0000 UTC m=+2589.373013266" Jul 30 20:07:58.130284 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:58.130255 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ff2f84-96877d8b4-bpxv8" Jul 30 20:07:58.130797 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:07:58.130336 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ff2f84-8d4bb8d4-7nd98" Jul 30 20:09:30.219308 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:30.219203 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-37abda-bb4d897bd-s5rrz"] Jul 30 20:09:30.219942 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:30.219459 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-37abda-bb4d897bd-s5rrz" podUID="152306ac-f28a-44ea-964e-47a936514063" containerName="payload-pre-processing" containerID="cri-o://0ae957de5348b176bb89aa7c63217b7b8c0a45053628ae2893bb7a09f7d49fa5" gracePeriod=30 Jul 30 20:09:30.475009 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:30.474955 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-37abda-bb4d897bd-s5rrz" Jul 30 20:09:30.588332 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:30.588302 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/152306ac-f28a-44ea-964e-47a936514063-plugins-config-volume\") pod \"152306ac-f28a-44ea-964e-47a936514063\" (UID: \"152306ac-f28a-44ea-964e-47a936514063\") " Jul 30 20:09:30.588477 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:30.588355 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g55xq\" (UniqueName: \"kubernetes.io/projected/152306ac-f28a-44ea-964e-47a936514063-kube-api-access-g55xq\") pod \"152306ac-f28a-44ea-964e-47a936514063\" (UID: \"152306ac-f28a-44ea-964e-47a936514063\") " Jul 30 20:09:30.588675 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:30.588653 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/152306ac-f28a-44ea-964e-47a936514063-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "152306ac-f28a-44ea-964e-47a936514063" (UID: "152306ac-f28a-44ea-964e-47a936514063"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 20:09:30.590374 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:30.590348 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/152306ac-f28a-44ea-964e-47a936514063-kube-api-access-g55xq" (OuterVolumeSpecName: "kube-api-access-g55xq") pod "152306ac-f28a-44ea-964e-47a936514063" (UID: "152306ac-f28a-44ea-964e-47a936514063"). InnerVolumeSpecName "kube-api-access-g55xq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:09:30.647331 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:30.647302 2556 generic.go:356] "Generic (PLEG): container finished" podID="152306ac-f28a-44ea-964e-47a936514063" containerID="0ae957de5348b176bb89aa7c63217b7b8c0a45053628ae2893bb7a09f7d49fa5" exitCode=0 Jul 30 20:09:30.647476 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:30.647346 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-37abda-bb4d897bd-s5rrz" Jul 30 20:09:30.647476 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:30.647412 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-37abda-bb4d897bd-s5rrz" event={"ID":"152306ac-f28a-44ea-964e-47a936514063","Type":"ContainerDied","Data":"0ae957de5348b176bb89aa7c63217b7b8c0a45053628ae2893bb7a09f7d49fa5"} Jul 30 20:09:30.647476 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:30.647466 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-37abda-bb4d897bd-s5rrz" event={"ID":"152306ac-f28a-44ea-964e-47a936514063","Type":"ContainerDied","Data":"b5db1ffd755e717137ce992ac710b1af4d95cd5967ca5d2e529534167c76087b"} Jul 30 20:09:30.647591 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:30.647489 2556 scope.go:117] "RemoveContainer" containerID="0ae957de5348b176bb89aa7c63217b7b8c0a45053628ae2893bb7a09f7d49fa5" Jul 30 20:09:30.656254 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:30.656236 2556 scope.go:117] "RemoveContainer" containerID="0ae957de5348b176bb89aa7c63217b7b8c0a45053628ae2893bb7a09f7d49fa5" Jul 30 20:09:30.656519 ip-10-0-129-105 kubenswrapper[2556]: E0730 20:09:30.656501 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ae957de5348b176bb89aa7c63217b7b8c0a45053628ae2893bb7a09f7d49fa5\": container with ID starting with 0ae957de5348b176bb89aa7c63217b7b8c0a45053628ae2893bb7a09f7d49fa5 not found: ID does not exist" containerID="0ae957de5348b176bb89aa7c63217b7b8c0a45053628ae2893bb7a09f7d49fa5" Jul 30 20:09:30.656586 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:30.656526 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ae957de5348b176bb89aa7c63217b7b8c0a45053628ae2893bb7a09f7d49fa5"} err="failed to get container status \"0ae957de5348b176bb89aa7c63217b7b8c0a45053628ae2893bb7a09f7d49fa5\": rpc error: code = NotFound desc = could not find container \"0ae957de5348b176bb89aa7c63217b7b8c0a45053628ae2893bb7a09f7d49fa5\": container with ID starting with 0ae957de5348b176bb89aa7c63217b7b8c0a45053628ae2893bb7a09f7d49fa5 not found: ID does not exist" Jul 30 20:09:30.668191 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:30.668171 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-37abda-bb4d897bd-s5rrz"] Jul 30 20:09:30.673795 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:30.673774 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-37abda-bb4d897bd-s5rrz"] Jul 30 20:09:30.689848 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:30.689827 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/152306ac-f28a-44ea-964e-47a936514063-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 20:09:30.689848 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:30.689847 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g55xq\" (UniqueName: \"kubernetes.io/projected/152306ac-f28a-44ea-964e-47a936514063-kube-api-access-g55xq\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 20:09:32.324488 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:32.324459 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="152306ac-f28a-44ea-964e-47a936514063" path="/var/lib/kubelet/pods/152306ac-f28a-44ea-964e-47a936514063/volumes" Jul 30 20:09:38.439147 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:38.439120 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-795qw_4cdd383f-6493-4d83-80fa-f4ce17e8416e/ovn-acl-logging/0.log" Jul 30 20:09:38.444358 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:38.444337 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-795qw_4cdd383f-6493-4d83-80fa-f4ce17e8416e/ovn-acl-logging/0.log" Jul 30 20:09:47.815620 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:47.815585 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ff2f84-96877d8b4-bpxv8"] Jul 30 20:09:47.816101 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:47.815813 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-ff2f84-96877d8b4-bpxv8" podUID="f5fc335d-498f-4be3-825e-fff2d0ee3b3d" containerName="payload-processing" containerID="cri-o://74f61c1a689cea8798c6447a61ebb5b6aaa493606cfa08358299615a8115e9c2" gracePeriod=30 Jul 30 20:09:47.823445 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:47.822620 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ff2f84-8d4bb8d4-7nd98"] Jul 30 20:09:47.823445 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:47.822835 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ff2f84-8d4bb8d4-7nd98" podUID="35bbbd1c-5a81-4e0f-bd7d-a1b20e95e1d3" containerName="payload-pre-processing" containerID="cri-o://559acaf1f685ada1c484e8d7396443e67aa6c81ba294c3a58bed17c64d6185d9" gracePeriod=30 Jul 30 20:09:48.079649 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:48.079581 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ff2f84-96877d8b4-bpxv8" Jul 30 20:09:48.113100 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:48.113071 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ff2f84-8d4bb8d4-7nd98" Jul 30 20:09:48.242490 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:48.242454 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-84hk2\" (UniqueName: \"kubernetes.io/projected/35bbbd1c-5a81-4e0f-bd7d-a1b20e95e1d3-kube-api-access-84hk2\") pod \"35bbbd1c-5a81-4e0f-bd7d-a1b20e95e1d3\" (UID: \"35bbbd1c-5a81-4e0f-bd7d-a1b20e95e1d3\") " Jul 30 20:09:48.242653 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:48.242519 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5fc335d-498f-4be3-825e-fff2d0ee3b3d-plugins-config-volume\") pod \"f5fc335d-498f-4be3-825e-fff2d0ee3b3d\" (UID: \"f5fc335d-498f-4be3-825e-fff2d0ee3b3d\") " Jul 30 20:09:48.242653 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:48.242546 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35bbbd1c-5a81-4e0f-bd7d-a1b20e95e1d3-plugins-config-volume\") pod \"35bbbd1c-5a81-4e0f-bd7d-a1b20e95e1d3\" (UID: \"35bbbd1c-5a81-4e0f-bd7d-a1b20e95e1d3\") " Jul 30 20:09:48.242653 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:48.242593 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-spmv6\" (UniqueName: \"kubernetes.io/projected/f5fc335d-498f-4be3-825e-fff2d0ee3b3d-kube-api-access-spmv6\") pod \"f5fc335d-498f-4be3-825e-fff2d0ee3b3d\" (UID: \"f5fc335d-498f-4be3-825e-fff2d0ee3b3d\") " Jul 30 20:09:48.242995 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:48.242949 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5fc335d-498f-4be3-825e-fff2d0ee3b3d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f5fc335d-498f-4be3-825e-fff2d0ee3b3d" (UID: "f5fc335d-498f-4be3-825e-fff2d0ee3b3d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 20:09:48.243128 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:48.242970 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35bbbd1c-5a81-4e0f-bd7d-a1b20e95e1d3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "35bbbd1c-5a81-4e0f-bd7d-a1b20e95e1d3" (UID: "35bbbd1c-5a81-4e0f-bd7d-a1b20e95e1d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 20:09:48.244575 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:48.244554 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35bbbd1c-5a81-4e0f-bd7d-a1b20e95e1d3-kube-api-access-84hk2" (OuterVolumeSpecName: "kube-api-access-84hk2") pod "35bbbd1c-5a81-4e0f-bd7d-a1b20e95e1d3" (UID: "35bbbd1c-5a81-4e0f-bd7d-a1b20e95e1d3"). InnerVolumeSpecName "kube-api-access-84hk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:09:48.244686 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:48.244653 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5fc335d-498f-4be3-825e-fff2d0ee3b3d-kube-api-access-spmv6" (OuterVolumeSpecName: "kube-api-access-spmv6") pod "f5fc335d-498f-4be3-825e-fff2d0ee3b3d" (UID: "f5fc335d-498f-4be3-825e-fff2d0ee3b3d"). InnerVolumeSpecName "kube-api-access-spmv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:09:48.343395 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:48.343366 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84hk2\" (UniqueName: \"kubernetes.io/projected/35bbbd1c-5a81-4e0f-bd7d-a1b20e95e1d3-kube-api-access-84hk2\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 20:09:48.343489 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:48.343431 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5fc335d-498f-4be3-825e-fff2d0ee3b3d-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 20:09:48.343489 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:48.343442 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35bbbd1c-5a81-4e0f-bd7d-a1b20e95e1d3-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 20:09:48.343489 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:48.343450 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spmv6\" (UniqueName: \"kubernetes.io/projected/f5fc335d-498f-4be3-825e-fff2d0ee3b3d-kube-api-access-spmv6\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 20:09:48.740463 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:48.740426 2556 generic.go:356] "Generic (PLEG): container finished" podID="f5fc335d-498f-4be3-825e-fff2d0ee3b3d" containerID="74f61c1a689cea8798c6447a61ebb5b6aaa493606cfa08358299615a8115e9c2" exitCode=0 Jul 30 20:09:48.740630 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:48.740471 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ff2f84-96877d8b4-bpxv8" Jul 30 20:09:48.740630 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:48.740520 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ff2f84-96877d8b4-bpxv8" event={"ID":"f5fc335d-498f-4be3-825e-fff2d0ee3b3d","Type":"ContainerDied","Data":"74f61c1a689cea8798c6447a61ebb5b6aaa493606cfa08358299615a8115e9c2"} Jul 30 20:09:48.740630 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:48.740568 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ff2f84-96877d8b4-bpxv8" event={"ID":"f5fc335d-498f-4be3-825e-fff2d0ee3b3d","Type":"ContainerDied","Data":"b5edeefcc4858b0cd616cfddaa8d0a4808604aa9720c7e971b0bb64dfb92b50f"} Jul 30 20:09:48.740630 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:48.740594 2556 scope.go:117] "RemoveContainer" containerID="74f61c1a689cea8798c6447a61ebb5b6aaa493606cfa08358299615a8115e9c2" Jul 30 20:09:48.741964 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:48.741937 2556 generic.go:356] "Generic (PLEG): container finished" podID="35bbbd1c-5a81-4e0f-bd7d-a1b20e95e1d3" containerID="559acaf1f685ada1c484e8d7396443e67aa6c81ba294c3a58bed17c64d6185d9" exitCode=0 Jul 30 20:09:48.742102 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:48.741978 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ff2f84-8d4bb8d4-7nd98" Jul 30 20:09:48.742102 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:48.742008 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ff2f84-8d4bb8d4-7nd98" event={"ID":"35bbbd1c-5a81-4e0f-bd7d-a1b20e95e1d3","Type":"ContainerDied","Data":"559acaf1f685ada1c484e8d7396443e67aa6c81ba294c3a58bed17c64d6185d9"} Jul 30 20:09:48.742102 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:48.742041 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ff2f84-8d4bb8d4-7nd98" event={"ID":"35bbbd1c-5a81-4e0f-bd7d-a1b20e95e1d3","Type":"ContainerDied","Data":"e5ed2d936e03745c07a4e33830de5840afbbbcaa8d32bd55863221c05b772bfe"} Jul 30 20:09:48.749339 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:48.749314 2556 scope.go:117] "RemoveContainer" containerID="74f61c1a689cea8798c6447a61ebb5b6aaa493606cfa08358299615a8115e9c2" Jul 30 20:09:48.749678 ip-10-0-129-105 kubenswrapper[2556]: E0730 20:09:48.749659 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74f61c1a689cea8798c6447a61ebb5b6aaa493606cfa08358299615a8115e9c2\": container with ID starting with 74f61c1a689cea8798c6447a61ebb5b6aaa493606cfa08358299615a8115e9c2 not found: ID does not exist" containerID="74f61c1a689cea8798c6447a61ebb5b6aaa493606cfa08358299615a8115e9c2" Jul 30 20:09:48.749777 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:48.749690 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74f61c1a689cea8798c6447a61ebb5b6aaa493606cfa08358299615a8115e9c2"} err="failed to get container status \"74f61c1a689cea8798c6447a61ebb5b6aaa493606cfa08358299615a8115e9c2\": rpc error: code = NotFound desc = could not find container \"74f61c1a689cea8798c6447a61ebb5b6aaa493606cfa08358299615a8115e9c2\": container with ID starting with 74f61c1a689cea8798c6447a61ebb5b6aaa493606cfa08358299615a8115e9c2 not found: ID does not exist" Jul 30 20:09:48.749777 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:48.749712 2556 scope.go:117] "RemoveContainer" containerID="559acaf1f685ada1c484e8d7396443e67aa6c81ba294c3a58bed17c64d6185d9" Jul 30 20:09:48.757416 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:48.757399 2556 scope.go:117] "RemoveContainer" containerID="559acaf1f685ada1c484e8d7396443e67aa6c81ba294c3a58bed17c64d6185d9" Jul 30 20:09:48.757673 ip-10-0-129-105 kubenswrapper[2556]: E0730 20:09:48.757654 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"559acaf1f685ada1c484e8d7396443e67aa6c81ba294c3a58bed17c64d6185d9\": container with ID starting with 559acaf1f685ada1c484e8d7396443e67aa6c81ba294c3a58bed17c64d6185d9 not found: ID does not exist" containerID="559acaf1f685ada1c484e8d7396443e67aa6c81ba294c3a58bed17c64d6185d9" Jul 30 20:09:48.757779 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:48.757681 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"559acaf1f685ada1c484e8d7396443e67aa6c81ba294c3a58bed17c64d6185d9"} err="failed to get container status \"559acaf1f685ada1c484e8d7396443e67aa6c81ba294c3a58bed17c64d6185d9\": rpc error: code = NotFound desc = could not find container \"559acaf1f685ada1c484e8d7396443e67aa6c81ba294c3a58bed17c64d6185d9\": container with ID starting with 559acaf1f685ada1c484e8d7396443e67aa6c81ba294c3a58bed17c64d6185d9 not found: ID does not exist" Jul 30 20:09:48.760585 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:48.760561 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ff2f84-8d4bb8d4-7nd98"] Jul 30 20:09:48.763874 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:48.763852 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ff2f84-8d4bb8d4-7nd98"] Jul 30 20:09:48.773764 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:48.773743 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ff2f84-96877d8b4-bpxv8"] Jul 30 20:09:48.775371 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:48.775353 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ff2f84-96877d8b4-bpxv8"] Jul 30 20:09:50.324976 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:50.324932 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35bbbd1c-5a81-4e0f-bd7d-a1b20e95e1d3" path="/var/lib/kubelet/pods/35bbbd1c-5a81-4e0f-bd7d-a1b20e95e1d3/volumes" Jul 30 20:09:50.325504 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:50.325467 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5fc335d-498f-4be3-825e-fff2d0ee3b3d" path="/var/lib/kubelet/pods/f5fc335d-498f-4be3-825e-fff2d0ee3b3d/volumes" Jul 30 20:09:59.713534 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:59.713493 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ec1ee8-748b978bfb-6nfzg"] Jul 30 20:09:59.713917 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:59.713904 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="152306ac-f28a-44ea-964e-47a936514063" containerName="payload-pre-processing" Jul 30 20:09:59.713967 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:59.713919 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="152306ac-f28a-44ea-964e-47a936514063" containerName="payload-pre-processing" Jul 30 20:09:59.713967 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:59.713927 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35bbbd1c-5a81-4e0f-bd7d-a1b20e95e1d3" containerName="payload-pre-processing" Jul 30 20:09:59.713967 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:59.713933 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="35bbbd1c-5a81-4e0f-bd7d-a1b20e95e1d3" containerName="payload-pre-processing" Jul 30 20:09:59.713967 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:59.713956 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5fc335d-498f-4be3-825e-fff2d0ee3b3d" containerName="payload-processing" Jul 30 20:09:59.713967 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:59.713963 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5fc335d-498f-4be3-825e-fff2d0ee3b3d" containerName="payload-processing" Jul 30 20:09:59.714116 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:59.714025 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5fc335d-498f-4be3-825e-fff2d0ee3b3d" containerName="payload-processing" Jul 30 20:09:59.714116 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:59.714034 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="35bbbd1c-5a81-4e0f-bd7d-a1b20e95e1d3" containerName="payload-pre-processing" Jul 30 20:09:59.714116 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:59.714040 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="152306ac-f28a-44ea-964e-47a936514063" containerName="payload-pre-processing" Jul 30 20:09:59.717141 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:59.717126 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec1ee8-748b978bfb-6nfzg" Jul 30 20:09:59.720615 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:59.720492 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-ec1ee8-dockercfg-7ll9m\"" Jul 30 20:09:59.721041 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:59.720662 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-ec1ee8\"" Jul 30 20:09:59.731856 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:59.731835 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ec1ee8-748b978bfb-6nfzg"] Jul 30 20:09:59.742745 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:59.742721 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ec1ee8-6f8b99d9c8-h6sdv"] Jul 30 20:09:59.743944 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:59.743917 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk9vl\" (UniqueName: \"kubernetes.io/projected/d90b7e6c-2481-4909-8e10-33a7be2da209-kube-api-access-dk9vl\") pod \"payload-pre-processing-e2e-api-a-ec1ee8-748b978bfb-6nfzg\" (UID: \"d90b7e6c-2481-4909-8e10-33a7be2da209\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec1ee8-748b978bfb-6nfzg" Jul 30 20:09:59.744048 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:59.743966 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d90b7e6c-2481-4909-8e10-33a7be2da209-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ec1ee8-748b978bfb-6nfzg\" (UID: \"d90b7e6c-2481-4909-8e10-33a7be2da209\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec1ee8-748b978bfb-6nfzg" Jul 30 20:09:59.745803 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:59.745784 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ec1ee8-6f8b99d9c8-h6sdv" Jul 30 20:09:59.756582 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:59.756561 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ec1ee8-6f8b99d9c8-h6sdv"] Jul 30 20:09:59.845438 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:59.845378 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfchd\" (UniqueName: \"kubernetes.io/projected/44a0293f-d6e9-4c37-b2b0-7c77e0aaa54a-kube-api-access-qfchd\") pod \"payload-processing-e2e-api-a-ec1ee8-6f8b99d9c8-h6sdv\" (UID: \"44a0293f-d6e9-4c37-b2b0-7c77e0aaa54a\") " pod="openshift-ingress/payload-processing-e2e-api-a-ec1ee8-6f8b99d9c8-h6sdv" Jul 30 20:09:59.845570 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:59.845446 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44a0293f-d6e9-4c37-b2b0-7c77e0aaa54a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ec1ee8-6f8b99d9c8-h6sdv\" (UID: \"44a0293f-d6e9-4c37-b2b0-7c77e0aaa54a\") " pod="openshift-ingress/payload-processing-e2e-api-a-ec1ee8-6f8b99d9c8-h6sdv" Jul 30 20:09:59.845570 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:59.845492 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk9vl\" (UniqueName: \"kubernetes.io/projected/d90b7e6c-2481-4909-8e10-33a7be2da209-kube-api-access-dk9vl\") pod \"payload-pre-processing-e2e-api-a-ec1ee8-748b978bfb-6nfzg\" (UID: \"d90b7e6c-2481-4909-8e10-33a7be2da209\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec1ee8-748b978bfb-6nfzg" Jul 30 20:09:59.845651 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:59.845618 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d90b7e6c-2481-4909-8e10-33a7be2da209-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ec1ee8-748b978bfb-6nfzg\" (UID: \"d90b7e6c-2481-4909-8e10-33a7be2da209\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec1ee8-748b978bfb-6nfzg" Jul 30 20:09:59.846269 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:59.846251 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d90b7e6c-2481-4909-8e10-33a7be2da209-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ec1ee8-748b978bfb-6nfzg\" (UID: \"d90b7e6c-2481-4909-8e10-33a7be2da209\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec1ee8-748b978bfb-6nfzg" Jul 30 20:09:59.853032 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:59.853006 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk9vl\" (UniqueName: \"kubernetes.io/projected/d90b7e6c-2481-4909-8e10-33a7be2da209-kube-api-access-dk9vl\") pod \"payload-pre-processing-e2e-api-a-ec1ee8-748b978bfb-6nfzg\" (UID: \"d90b7e6c-2481-4909-8e10-33a7be2da209\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec1ee8-748b978bfb-6nfzg" Jul 30 20:09:59.947166 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:59.947136 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfchd\" (UniqueName: \"kubernetes.io/projected/44a0293f-d6e9-4c37-b2b0-7c77e0aaa54a-kube-api-access-qfchd\") pod \"payload-processing-e2e-api-a-ec1ee8-6f8b99d9c8-h6sdv\" (UID: \"44a0293f-d6e9-4c37-b2b0-7c77e0aaa54a\") " pod="openshift-ingress/payload-processing-e2e-api-a-ec1ee8-6f8b99d9c8-h6sdv" Jul 30 20:09:59.947166 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:59.947171 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44a0293f-d6e9-4c37-b2b0-7c77e0aaa54a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ec1ee8-6f8b99d9c8-h6sdv\" (UID: \"44a0293f-d6e9-4c37-b2b0-7c77e0aaa54a\") " pod="openshift-ingress/payload-processing-e2e-api-a-ec1ee8-6f8b99d9c8-h6sdv" Jul 30 20:09:59.947820 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:59.947803 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44a0293f-d6e9-4c37-b2b0-7c77e0aaa54a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ec1ee8-6f8b99d9c8-h6sdv\" (UID: \"44a0293f-d6e9-4c37-b2b0-7c77e0aaa54a\") " pod="openshift-ingress/payload-processing-e2e-api-a-ec1ee8-6f8b99d9c8-h6sdv" Jul 30 20:09:59.954674 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:09:59.954650 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfchd\" (UniqueName: \"kubernetes.io/projected/44a0293f-d6e9-4c37-b2b0-7c77e0aaa54a-kube-api-access-qfchd\") pod \"payload-processing-e2e-api-a-ec1ee8-6f8b99d9c8-h6sdv\" (UID: \"44a0293f-d6e9-4c37-b2b0-7c77e0aaa54a\") " pod="openshift-ingress/payload-processing-e2e-api-a-ec1ee8-6f8b99d9c8-h6sdv" Jul 30 20:10:00.029603 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:10:00.029510 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec1ee8-748b978bfb-6nfzg" Jul 30 20:10:00.056345 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:10:00.056320 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ec1ee8-6f8b99d9c8-h6sdv" Jul 30 20:10:00.177072 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:10:00.177045 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ec1ee8-748b978bfb-6nfzg"] Jul 30 20:10:00.179528 ip-10-0-129-105 kubenswrapper[2556]: W0730 20:10:00.179503 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd90b7e6c_2481_4909_8e10_33a7be2da209.slice/crio-908b9abeef818087ac3275d9fe0a2d23c35c8f09166211975ac8b8f6e349606b WatchSource:0}: Error finding container 908b9abeef818087ac3275d9fe0a2d23c35c8f09166211975ac8b8f6e349606b: Status 404 returned error can't find the container with id 908b9abeef818087ac3275d9fe0a2d23c35c8f09166211975ac8b8f6e349606b Jul 30 20:10:00.412390 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:10:00.412344 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ec1ee8-6f8b99d9c8-h6sdv"] Jul 30 20:10:00.413784 ip-10-0-129-105 kubenswrapper[2556]: W0730 20:10:00.413760 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44a0293f_d6e9_4c37_b2b0_7c77e0aaa54a.slice/crio-e6d3c36d5b8fa433f773817e78f34949fa56db505ccde5c465b3e1cfde826c97 WatchSource:0}: Error finding container e6d3c36d5b8fa433f773817e78f34949fa56db505ccde5c465b3e1cfde826c97: Status 404 returned error can't find the container with id e6d3c36d5b8fa433f773817e78f34949fa56db505ccde5c465b3e1cfde826c97 Jul 30 20:10:00.805890 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:10:00.805783 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ec1ee8-6f8b99d9c8-h6sdv" event={"ID":"44a0293f-d6e9-4c37-b2b0-7c77e0aaa54a","Type":"ContainerStarted","Data":"cd96ccee755d7614e83508aa1e680510a011b6520be6f9a3059fea7bdb3d412d"} Jul 30 20:10:00.805890 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:10:00.805847 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ec1ee8-6f8b99d9c8-h6sdv" event={"ID":"44a0293f-d6e9-4c37-b2b0-7c77e0aaa54a","Type":"ContainerStarted","Data":"e6d3c36d5b8fa433f773817e78f34949fa56db505ccde5c465b3e1cfde826c97"} Jul 30 20:10:00.806369 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:10:00.805965 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-ec1ee8-6f8b99d9c8-h6sdv" Jul 30 20:10:00.807404 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:10:00.807362 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec1ee8-748b978bfb-6nfzg" event={"ID":"d90b7e6c-2481-4909-8e10-33a7be2da209","Type":"ContainerStarted","Data":"c517889223f5d50b13a2dfd9ad63da8da18afdbcf7ca5fc73c41d9b7c71d5c5f"} Jul 30 20:10:00.807525 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:10:00.807430 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec1ee8-748b978bfb-6nfzg" event={"ID":"d90b7e6c-2481-4909-8e10-33a7be2da209","Type":"ContainerStarted","Data":"908b9abeef818087ac3275d9fe0a2d23c35c8f09166211975ac8b8f6e349606b"} Jul 30 20:10:00.807525 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:10:00.807467 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec1ee8-748b978bfb-6nfzg" Jul 30 20:10:00.824544 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:10:00.824500 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-ec1ee8-6f8b99d9c8-h6sdv" podStartSLOduration=1.824482894 podStartE2EDuration="1.824482894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:09:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:10:00.821191349 +0000 UTC m=+2723.040750265" watchObservedRunningTime="2026-07-30 20:10:00.824482894 +0000 UTC m=+2723.044041808" Jul 30 20:10:00.837777 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:10:00.837729 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec1ee8-748b978bfb-6nfzg" podStartSLOduration=1.837713208 podStartE2EDuration="1.837713208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:09:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:10:00.836691488 +0000 UTC m=+2723.056250405" watchObservedRunningTime="2026-07-30 20:10:00.837713208 +0000 UTC m=+2723.057272177" Jul 30 20:10:11.814735 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:10:11.814699 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec1ee8-748b978bfb-6nfzg" Jul 30 20:10:11.815206 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:10:11.814872 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-ec1ee8-6f8b99d9c8-h6sdv" Jul 30 20:10:28.814187 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:10:28.814153 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-18459c-768bbfdb88-25j9q"] Jul 30 20:10:28.817785 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:10:28.817767 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18459c-768bbfdb88-25j9q" Jul 30 20:10:28.821427 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:10:28.821404 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-18459c-dockercfg-cbhzc\"" Jul 30 20:10:28.821540 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:10:28.821438 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-18459c\"" Jul 30 20:10:28.830831 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:10:28.830808 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-18459c-768bbfdb88-25j9q"] Jul 30 20:10:28.833409 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:10:28.833373 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-18459c-69574c7858-47dk6"] Jul 30 20:10:28.837056 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:10:28.837039 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-18459c-69574c7858-47dk6" Jul 30 20:10:28.846592 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:10:28.846572 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-18459c-69574c7858-47dk6"] Jul 30 20:10:29.001829 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:10:29.001799 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgjkm\" (UniqueName: \"kubernetes.io/projected/aabcfdfd-ce62-4e66-a4fd-acd3cc64e7dd-kube-api-access-lgjkm\") pod \"payload-pre-processing-e2e-api-b-18459c-768bbfdb88-25j9q\" (UID: \"aabcfdfd-ce62-4e66-a4fd-acd3cc64e7dd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-18459c-768bbfdb88-25j9q" Jul 30 20:10:29.002002 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:10:29.001866 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9h7d\" (UniqueName: \"kubernetes.io/projected/cc727233-80e3-4b89-94f6-e7ffea06c05d-kube-api-access-b9h7d\") pod \"payload-processing-e2e-api-b-18459c-69574c7858-47dk6\" (UID: \"cc727233-80e3-4b89-94f6-e7ffea06c05d\") " pod="openshift-ingress/payload-processing-e2e-api-b-18459c-69574c7858-47dk6" Jul 30 20:10:29.002002 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:10:29.001889 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aabcfdfd-ce62-4e66-a4fd-acd3cc64e7dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-18459c-768bbfdb88-25j9q\" (UID: \"aabcfdfd-ce62-4e66-a4fd-acd3cc64e7dd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-18459c-768bbfdb88-25j9q" Jul 30 20:10:29.002002 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:10:29.001918 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc727233-80e3-4b89-94f6-e7ffea06c05d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-18459c-69574c7858-47dk6\" (UID: \"cc727233-80e3-4b89-94f6-e7ffea06c05d\") " pod="openshift-ingress/payload-processing-e2e-api-b-18459c-69574c7858-47dk6" Jul 30 20:10:29.102660 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:10:29.102581 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9h7d\" (UniqueName: \"kubernetes.io/projected/cc727233-80e3-4b89-94f6-e7ffea06c05d-kube-api-access-b9h7d\") pod \"payload-processing-e2e-api-b-18459c-69574c7858-47dk6\" (UID: \"cc727233-80e3-4b89-94f6-e7ffea06c05d\") " pod="openshift-ingress/payload-processing-e2e-api-b-18459c-69574c7858-47dk6" Jul 30 20:10:29.102660 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:10:29.102619 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aabcfdfd-ce62-4e66-a4fd-acd3cc64e7dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-18459c-768bbfdb88-25j9q\" (UID: \"aabcfdfd-ce62-4e66-a4fd-acd3cc64e7dd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-18459c-768bbfdb88-25j9q" Jul 30 20:10:29.102660 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:10:29.102650 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc727233-80e3-4b89-94f6-e7ffea06c05d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-18459c-69574c7858-47dk6\" (UID: \"cc727233-80e3-4b89-94f6-e7ffea06c05d\") " pod="openshift-ingress/payload-processing-e2e-api-b-18459c-69574c7858-47dk6" Jul 30 20:10:29.102874 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:10:29.102737 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgjkm\" (UniqueName: \"kubernetes.io/projected/aabcfdfd-ce62-4e66-a4fd-acd3cc64e7dd-kube-api-access-lgjkm\") pod \"payload-pre-processing-e2e-api-b-18459c-768bbfdb88-25j9q\" (UID: \"aabcfdfd-ce62-4e66-a4fd-acd3cc64e7dd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-18459c-768bbfdb88-25j9q" Jul 30 20:10:29.103347 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:10:29.103326 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aabcfdfd-ce62-4e66-a4fd-acd3cc64e7dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-18459c-768bbfdb88-25j9q\" (UID: \"aabcfdfd-ce62-4e66-a4fd-acd3cc64e7dd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-18459c-768bbfdb88-25j9q" Jul 30 20:10:29.103402 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:10:29.103354 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc727233-80e3-4b89-94f6-e7ffea06c05d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-18459c-69574c7858-47dk6\" (UID: \"cc727233-80e3-4b89-94f6-e7ffea06c05d\") " pod="openshift-ingress/payload-processing-e2e-api-b-18459c-69574c7858-47dk6" Jul 30 20:10:29.110722 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:10:29.110699 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9h7d\" (UniqueName: \"kubernetes.io/projected/cc727233-80e3-4b89-94f6-e7ffea06c05d-kube-api-access-b9h7d\") pod \"payload-processing-e2e-api-b-18459c-69574c7858-47dk6\" (UID: \"cc727233-80e3-4b89-94f6-e7ffea06c05d\") " pod="openshift-ingress/payload-processing-e2e-api-b-18459c-69574c7858-47dk6" Jul 30 20:10:29.110845 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:10:29.110814 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgjkm\" (UniqueName: \"kubernetes.io/projected/aabcfdfd-ce62-4e66-a4fd-acd3cc64e7dd-kube-api-access-lgjkm\") pod \"payload-pre-processing-e2e-api-b-18459c-768bbfdb88-25j9q\" (UID: \"aabcfdfd-ce62-4e66-a4fd-acd3cc64e7dd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-18459c-768bbfdb88-25j9q" Jul 30 20:10:29.129644 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:10:29.129621 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18459c-768bbfdb88-25j9q" Jul 30 20:10:29.148470 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:10:29.148450 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-18459c-69574c7858-47dk6" Jul 30 20:10:29.483994 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:10:29.483972 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-18459c-768bbfdb88-25j9q"] Jul 30 20:10:29.486237 ip-10-0-129-105 kubenswrapper[2556]: W0730 20:10:29.486205 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaabcfdfd_ce62_4e66_a4fd_acd3cc64e7dd.slice/crio-7ca3134e23cd49ef77d59f003b545c72872f0c8bd2f6e90eb572604b0a3f94e3 WatchSource:0}: Error finding container 7ca3134e23cd49ef77d59f003b545c72872f0c8bd2f6e90eb572604b0a3f94e3: Status 404 returned error can't find the container with id 7ca3134e23cd49ef77d59f003b545c72872f0c8bd2f6e90eb572604b0a3f94e3 Jul 30 20:10:29.516820 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:10:29.516797 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-18459c-69574c7858-47dk6"] Jul 30 20:10:29.521523 ip-10-0-129-105 kubenswrapper[2556]: W0730 20:10:29.521497 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc727233_80e3_4b89_94f6_e7ffea06c05d.slice/crio-37a9642ffede214d079cbe62d4ddc78ecba3f99383914149e23a277f30a9398a WatchSource:0}: Error finding container 37a9642ffede214d079cbe62d4ddc78ecba3f99383914149e23a277f30a9398a: Status 404 returned error can't find the container with id 37a9642ffede214d079cbe62d4ddc78ecba3f99383914149e23a277f30a9398a Jul 30 20:10:29.960901 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:10:29.960864 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18459c-768bbfdb88-25j9q" event={"ID":"aabcfdfd-ce62-4e66-a4fd-acd3cc64e7dd","Type":"ContainerStarted","Data":"1cd18a16b2dcb3f7e86dd07211979ae731455c9847313e2338a0ff2e6a8834d2"} Jul 30 20:10:29.961351 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:10:29.960932 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18459c-768bbfdb88-25j9q" Jul 30 20:10:29.961351 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:10:29.960959 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18459c-768bbfdb88-25j9q" event={"ID":"aabcfdfd-ce62-4e66-a4fd-acd3cc64e7dd","Type":"ContainerStarted","Data":"7ca3134e23cd49ef77d59f003b545c72872f0c8bd2f6e90eb572604b0a3f94e3"} Jul 30 20:10:29.962498 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:10:29.962463 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-18459c-69574c7858-47dk6" event={"ID":"cc727233-80e3-4b89-94f6-e7ffea06c05d","Type":"ContainerStarted","Data":"52620863b40c6532af69c3596ca95bfd43c84a1a819923d6ab7942c07909e1c4"} Jul 30 20:10:29.962635 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:10:29.962520 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-18459c-69574c7858-47dk6" event={"ID":"cc727233-80e3-4b89-94f6-e7ffea06c05d","Type":"ContainerStarted","Data":"37a9642ffede214d079cbe62d4ddc78ecba3f99383914149e23a277f30a9398a"} Jul 30 20:10:29.962635 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:10:29.962560 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-18459c-69574c7858-47dk6" Jul 30 20:10:29.978767 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:10:29.978728 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18459c-768bbfdb88-25j9q" podStartSLOduration=1.978714417 podStartE2EDuration="1.978714417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:10:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:10:29.977017055 +0000 UTC m=+2752.196575968" watchObservedRunningTime="2026-07-30 20:10:29.978714417 +0000 UTC m=+2752.198273370" Jul 30 20:10:29.994233 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:10:29.994191 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-18459c-69574c7858-47dk6" podStartSLOduration=1.9941783640000001 podStartE2EDuration="1.994178364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:10:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:10:29.993073047 +0000 UTC m=+2752.212631962" watchObservedRunningTime="2026-07-30 20:10:29.994178364 +0000 UTC m=+2752.213737277" Jul 30 20:10:40.969289 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:10:40.969259 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18459c-768bbfdb88-25j9q" Jul 30 20:10:40.969794 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:10:40.969338 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-18459c-69574c7858-47dk6" Jul 30 20:11:10.302167 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:10.302078 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ec1ee8-6f8b99d9c8-h6sdv"] Jul 30 20:11:10.302721 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:10.302280 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-ec1ee8-6f8b99d9c8-h6sdv" podUID="44a0293f-d6e9-4c37-b2b0-7c77e0aaa54a" containerName="payload-processing" containerID="cri-o://cd96ccee755d7614e83508aa1e680510a011b6520be6f9a3059fea7bdb3d412d" gracePeriod=30 Jul 30 20:11:10.308109 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:10.308071 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ec1ee8-748b978bfb-6nfzg"] Jul 30 20:11:10.308622 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:10.308544 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec1ee8-748b978bfb-6nfzg" podUID="d90b7e6c-2481-4909-8e10-33a7be2da209" containerName="payload-pre-processing" containerID="cri-o://c517889223f5d50b13a2dfd9ad63da8da18afdbcf7ca5fc73c41d9b7c71d5c5f" gracePeriod=30 Jul 30 20:11:10.594784 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:10.594761 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec1ee8-748b978bfb-6nfzg" Jul 30 20:11:10.597594 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:10.597574 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ec1ee8-6f8b99d9c8-h6sdv" Jul 30 20:11:10.669367 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:10.669339 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44a0293f-d6e9-4c37-b2b0-7c77e0aaa54a-plugins-config-volume\") pod \"44a0293f-d6e9-4c37-b2b0-7c77e0aaa54a\" (UID: \"44a0293f-d6e9-4c37-b2b0-7c77e0aaa54a\") " Jul 30 20:11:10.669534 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:10.669441 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dk9vl\" (UniqueName: \"kubernetes.io/projected/d90b7e6c-2481-4909-8e10-33a7be2da209-kube-api-access-dk9vl\") pod \"d90b7e6c-2481-4909-8e10-33a7be2da209\" (UID: \"d90b7e6c-2481-4909-8e10-33a7be2da209\") " Jul 30 20:11:10.669534 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:10.669488 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfchd\" (UniqueName: \"kubernetes.io/projected/44a0293f-d6e9-4c37-b2b0-7c77e0aaa54a-kube-api-access-qfchd\") pod \"44a0293f-d6e9-4c37-b2b0-7c77e0aaa54a\" (UID: \"44a0293f-d6e9-4c37-b2b0-7c77e0aaa54a\") " Jul 30 20:11:10.669617 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:10.669571 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d90b7e6c-2481-4909-8e10-33a7be2da209-plugins-config-volume\") pod \"d90b7e6c-2481-4909-8e10-33a7be2da209\" (UID: \"d90b7e6c-2481-4909-8e10-33a7be2da209\") " Jul 30 20:11:10.669724 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:10.669696 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44a0293f-d6e9-4c37-b2b0-7c77e0aaa54a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "44a0293f-d6e9-4c37-b2b0-7c77e0aaa54a" (UID: "44a0293f-d6e9-4c37-b2b0-7c77e0aaa54a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 20:11:10.669940 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:10.669918 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44a0293f-d6e9-4c37-b2b0-7c77e0aaa54a-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 20:11:10.669940 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:10.669935 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d90b7e6c-2481-4909-8e10-33a7be2da209-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d90b7e6c-2481-4909-8e10-33a7be2da209" (UID: "d90b7e6c-2481-4909-8e10-33a7be2da209"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 20:11:10.671720 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:10.671695 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d90b7e6c-2481-4909-8e10-33a7be2da209-kube-api-access-dk9vl" (OuterVolumeSpecName: "kube-api-access-dk9vl") pod "d90b7e6c-2481-4909-8e10-33a7be2da209" (UID: "d90b7e6c-2481-4909-8e10-33a7be2da209"). InnerVolumeSpecName "kube-api-access-dk9vl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:11:10.671827 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:10.671804 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44a0293f-d6e9-4c37-b2b0-7c77e0aaa54a-kube-api-access-qfchd" (OuterVolumeSpecName: "kube-api-access-qfchd") pod "44a0293f-d6e9-4c37-b2b0-7c77e0aaa54a" (UID: "44a0293f-d6e9-4c37-b2b0-7c77e0aaa54a"). InnerVolumeSpecName "kube-api-access-qfchd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:11:10.770983 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:10.770951 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk9vl\" (UniqueName: \"kubernetes.io/projected/d90b7e6c-2481-4909-8e10-33a7be2da209-kube-api-access-dk9vl\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 20:11:10.770983 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:10.770977 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfchd\" (UniqueName: \"kubernetes.io/projected/44a0293f-d6e9-4c37-b2b0-7c77e0aaa54a-kube-api-access-qfchd\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 20:11:10.770983 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:10.770986 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d90b7e6c-2481-4909-8e10-33a7be2da209-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 20:11:11.174537 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:11.174503 2556 generic.go:356] "Generic (PLEG): container finished" podID="44a0293f-d6e9-4c37-b2b0-7c77e0aaa54a" containerID="cd96ccee755d7614e83508aa1e680510a011b6520be6f9a3059fea7bdb3d412d" exitCode=0 Jul 30 20:11:11.174537 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:11.174546 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ec1ee8-6f8b99d9c8-h6sdv" Jul 30 20:11:11.174794 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:11.174591 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ec1ee8-6f8b99d9c8-h6sdv" event={"ID":"44a0293f-d6e9-4c37-b2b0-7c77e0aaa54a","Type":"ContainerDied","Data":"cd96ccee755d7614e83508aa1e680510a011b6520be6f9a3059fea7bdb3d412d"} Jul 30 20:11:11.174794 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:11.174640 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ec1ee8-6f8b99d9c8-h6sdv" event={"ID":"44a0293f-d6e9-4c37-b2b0-7c77e0aaa54a","Type":"ContainerDied","Data":"e6d3c36d5b8fa433f773817e78f34949fa56db505ccde5c465b3e1cfde826c97"} Jul 30 20:11:11.174794 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:11.174664 2556 scope.go:117] "RemoveContainer" containerID="cd96ccee755d7614e83508aa1e680510a011b6520be6f9a3059fea7bdb3d412d" Jul 30 20:11:11.175983 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:11.175952 2556 generic.go:356] "Generic (PLEG): container finished" podID="d90b7e6c-2481-4909-8e10-33a7be2da209" containerID="c517889223f5d50b13a2dfd9ad63da8da18afdbcf7ca5fc73c41d9b7c71d5c5f" exitCode=0 Jul 30 20:11:11.176109 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:11.175991 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec1ee8-748b978bfb-6nfzg" event={"ID":"d90b7e6c-2481-4909-8e10-33a7be2da209","Type":"ContainerDied","Data":"c517889223f5d50b13a2dfd9ad63da8da18afdbcf7ca5fc73c41d9b7c71d5c5f"} Jul 30 20:11:11.176109 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:11.176003 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec1ee8-748b978bfb-6nfzg" Jul 30 20:11:11.176109 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:11.176024 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec1ee8-748b978bfb-6nfzg" event={"ID":"d90b7e6c-2481-4909-8e10-33a7be2da209","Type":"ContainerDied","Data":"908b9abeef818087ac3275d9fe0a2d23c35c8f09166211975ac8b8f6e349606b"} Jul 30 20:11:11.185204 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:11.185189 2556 scope.go:117] "RemoveContainer" containerID="cd96ccee755d7614e83508aa1e680510a011b6520be6f9a3059fea7bdb3d412d" Jul 30 20:11:11.185589 ip-10-0-129-105 kubenswrapper[2556]: E0730 20:11:11.185565 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd96ccee755d7614e83508aa1e680510a011b6520be6f9a3059fea7bdb3d412d\": container with ID starting with cd96ccee755d7614e83508aa1e680510a011b6520be6f9a3059fea7bdb3d412d not found: ID does not exist" containerID="cd96ccee755d7614e83508aa1e680510a011b6520be6f9a3059fea7bdb3d412d" Jul 30 20:11:11.185707 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:11.185596 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd96ccee755d7614e83508aa1e680510a011b6520be6f9a3059fea7bdb3d412d"} err="failed to get container status \"cd96ccee755d7614e83508aa1e680510a011b6520be6f9a3059fea7bdb3d412d\": rpc error: code = NotFound desc = could not find container \"cd96ccee755d7614e83508aa1e680510a011b6520be6f9a3059fea7bdb3d412d\": container with ID starting with cd96ccee755d7614e83508aa1e680510a011b6520be6f9a3059fea7bdb3d412d not found: ID does not exist" Jul 30 20:11:11.185707 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:11.185612 2556 scope.go:117] "RemoveContainer" containerID="c517889223f5d50b13a2dfd9ad63da8da18afdbcf7ca5fc73c41d9b7c71d5c5f" Jul 30 20:11:11.195710 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:11.195696 2556 scope.go:117] "RemoveContainer" containerID="c517889223f5d50b13a2dfd9ad63da8da18afdbcf7ca5fc73c41d9b7c71d5c5f" Jul 30 20:11:11.195940 ip-10-0-129-105 kubenswrapper[2556]: E0730 20:11:11.195921 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c517889223f5d50b13a2dfd9ad63da8da18afdbcf7ca5fc73c41d9b7c71d5c5f\": container with ID starting with c517889223f5d50b13a2dfd9ad63da8da18afdbcf7ca5fc73c41d9b7c71d5c5f not found: ID does not exist" containerID="c517889223f5d50b13a2dfd9ad63da8da18afdbcf7ca5fc73c41d9b7c71d5c5f" Jul 30 20:11:11.196002 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:11.195945 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c517889223f5d50b13a2dfd9ad63da8da18afdbcf7ca5fc73c41d9b7c71d5c5f"} err="failed to get container status \"c517889223f5d50b13a2dfd9ad63da8da18afdbcf7ca5fc73c41d9b7c71d5c5f\": rpc error: code = NotFound desc = could not find container \"c517889223f5d50b13a2dfd9ad63da8da18afdbcf7ca5fc73c41d9b7c71d5c5f\": container with ID starting with c517889223f5d50b13a2dfd9ad63da8da18afdbcf7ca5fc73c41d9b7c71d5c5f not found: ID does not exist" Jul 30 20:11:11.199470 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:11.199449 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ec1ee8-6f8b99d9c8-h6sdv"] Jul 30 20:11:11.201504 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:11.201483 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ec1ee8-6f8b99d9c8-h6sdv"] Jul 30 20:11:11.210281 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:11.210260 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ec1ee8-748b978bfb-6nfzg"] Jul 30 20:11:11.214236 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:11.214218 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ec1ee8-748b978bfb-6nfzg"] Jul 30 20:11:12.324800 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:12.324764 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44a0293f-d6e9-4c37-b2b0-7c77e0aaa54a" path="/var/lib/kubelet/pods/44a0293f-d6e9-4c37-b2b0-7c77e0aaa54a/volumes" Jul 30 20:11:12.325177 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:12.325124 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d90b7e6c-2481-4909-8e10-33a7be2da209" path="/var/lib/kubelet/pods/d90b7e6c-2481-4909-8e10-33a7be2da209/volumes" Jul 30 20:11:32.637125 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:32.637089 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-18459c-69574c7858-47dk6"] Jul 30 20:11:32.637665 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:32.637355 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-18459c-69574c7858-47dk6" podUID="cc727233-80e3-4b89-94f6-e7ffea06c05d" containerName="payload-processing" containerID="cri-o://52620863b40c6532af69c3596ca95bfd43c84a1a819923d6ab7942c07909e1c4" gracePeriod=30 Jul 30 20:11:32.642642 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:32.642614 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-18459c-768bbfdb88-25j9q"] Jul 30 20:11:32.643011 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:32.642985 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18459c-768bbfdb88-25j9q" podUID="aabcfdfd-ce62-4e66-a4fd-acd3cc64e7dd" containerName="payload-pre-processing" containerID="cri-o://1cd18a16b2dcb3f7e86dd07211979ae731455c9847313e2338a0ff2e6a8834d2" gracePeriod=30 Jul 30 20:11:32.907845 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:32.907816 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-18459c-69574c7858-47dk6" Jul 30 20:11:32.920767 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:32.920746 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18459c-768bbfdb88-25j9q" Jul 30 20:11:32.973196 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:32.973175 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgjkm\" (UniqueName: \"kubernetes.io/projected/aabcfdfd-ce62-4e66-a4fd-acd3cc64e7dd-kube-api-access-lgjkm\") pod \"aabcfdfd-ce62-4e66-a4fd-acd3cc64e7dd\" (UID: \"aabcfdfd-ce62-4e66-a4fd-acd3cc64e7dd\") " Jul 30 20:11:32.973312 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:32.973223 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aabcfdfd-ce62-4e66-a4fd-acd3cc64e7dd-plugins-config-volume\") pod \"aabcfdfd-ce62-4e66-a4fd-acd3cc64e7dd\" (UID: \"aabcfdfd-ce62-4e66-a4fd-acd3cc64e7dd\") " Jul 30 20:11:32.973312 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:32.973309 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9h7d\" (UniqueName: \"kubernetes.io/projected/cc727233-80e3-4b89-94f6-e7ffea06c05d-kube-api-access-b9h7d\") pod \"cc727233-80e3-4b89-94f6-e7ffea06c05d\" (UID: \"cc727233-80e3-4b89-94f6-e7ffea06c05d\") " Jul 30 20:11:32.973403 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:32.973337 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc727233-80e3-4b89-94f6-e7ffea06c05d-plugins-config-volume\") pod \"cc727233-80e3-4b89-94f6-e7ffea06c05d\" (UID: \"cc727233-80e3-4b89-94f6-e7ffea06c05d\") " Jul 30 20:11:32.973701 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:32.973676 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aabcfdfd-ce62-4e66-a4fd-acd3cc64e7dd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "aabcfdfd-ce62-4e66-a4fd-acd3cc64e7dd" (UID: "aabcfdfd-ce62-4e66-a4fd-acd3cc64e7dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 20:11:32.973783 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:32.973724 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc727233-80e3-4b89-94f6-e7ffea06c05d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cc727233-80e3-4b89-94f6-e7ffea06c05d" (UID: "cc727233-80e3-4b89-94f6-e7ffea06c05d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 20:11:32.975250 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:32.975227 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aabcfdfd-ce62-4e66-a4fd-acd3cc64e7dd-kube-api-access-lgjkm" (OuterVolumeSpecName: "kube-api-access-lgjkm") pod "aabcfdfd-ce62-4e66-a4fd-acd3cc64e7dd" (UID: "aabcfdfd-ce62-4e66-a4fd-acd3cc64e7dd"). InnerVolumeSpecName "kube-api-access-lgjkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:11:32.975353 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:32.975256 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc727233-80e3-4b89-94f6-e7ffea06c05d-kube-api-access-b9h7d" (OuterVolumeSpecName: "kube-api-access-b9h7d") pod "cc727233-80e3-4b89-94f6-e7ffea06c05d" (UID: "cc727233-80e3-4b89-94f6-e7ffea06c05d"). InnerVolumeSpecName "kube-api-access-b9h7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:11:33.074516 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:33.074482 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9h7d\" (UniqueName: \"kubernetes.io/projected/cc727233-80e3-4b89-94f6-e7ffea06c05d-kube-api-access-b9h7d\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 20:11:33.074516 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:33.074510 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc727233-80e3-4b89-94f6-e7ffea06c05d-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 20:11:33.074516 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:33.074521 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgjkm\" (UniqueName: \"kubernetes.io/projected/aabcfdfd-ce62-4e66-a4fd-acd3cc64e7dd-kube-api-access-lgjkm\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 20:11:33.074732 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:33.074532 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aabcfdfd-ce62-4e66-a4fd-acd3cc64e7dd-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 20:11:33.292281 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:33.292195 2556 generic.go:356] "Generic (PLEG): container finished" podID="cc727233-80e3-4b89-94f6-e7ffea06c05d" containerID="52620863b40c6532af69c3596ca95bfd43c84a1a819923d6ab7942c07909e1c4" exitCode=0 Jul 30 20:11:33.292281 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:33.292235 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-18459c-69574c7858-47dk6" Jul 30 20:11:33.292545 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:33.292289 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-18459c-69574c7858-47dk6" event={"ID":"cc727233-80e3-4b89-94f6-e7ffea06c05d","Type":"ContainerDied","Data":"52620863b40c6532af69c3596ca95bfd43c84a1a819923d6ab7942c07909e1c4"} Jul 30 20:11:33.292545 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:33.292345 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-18459c-69574c7858-47dk6" event={"ID":"cc727233-80e3-4b89-94f6-e7ffea06c05d","Type":"ContainerDied","Data":"37a9642ffede214d079cbe62d4ddc78ecba3f99383914149e23a277f30a9398a"} Jul 30 20:11:33.292545 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:33.292377 2556 scope.go:117] "RemoveContainer" containerID="52620863b40c6532af69c3596ca95bfd43c84a1a819923d6ab7942c07909e1c4" Jul 30 20:11:33.293554 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:33.293528 2556 generic.go:356] "Generic (PLEG): container finished" podID="aabcfdfd-ce62-4e66-a4fd-acd3cc64e7dd" containerID="1cd18a16b2dcb3f7e86dd07211979ae731455c9847313e2338a0ff2e6a8834d2" exitCode=0 Jul 30 20:11:33.293676 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:33.293570 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18459c-768bbfdb88-25j9q" Jul 30 20:11:33.293676 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:33.293604 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18459c-768bbfdb88-25j9q" event={"ID":"aabcfdfd-ce62-4e66-a4fd-acd3cc64e7dd","Type":"ContainerDied","Data":"1cd18a16b2dcb3f7e86dd07211979ae731455c9847313e2338a0ff2e6a8834d2"} Jul 30 20:11:33.293676 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:33.293656 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18459c-768bbfdb88-25j9q" event={"ID":"aabcfdfd-ce62-4e66-a4fd-acd3cc64e7dd","Type":"ContainerDied","Data":"7ca3134e23cd49ef77d59f003b545c72872f0c8bd2f6e90eb572604b0a3f94e3"} Jul 30 20:11:33.302733 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:33.302716 2556 scope.go:117] "RemoveContainer" containerID="52620863b40c6532af69c3596ca95bfd43c84a1a819923d6ab7942c07909e1c4" Jul 30 20:11:33.302968 ip-10-0-129-105 kubenswrapper[2556]: E0730 20:11:33.302951 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52620863b40c6532af69c3596ca95bfd43c84a1a819923d6ab7942c07909e1c4\": container with ID starting with 52620863b40c6532af69c3596ca95bfd43c84a1a819923d6ab7942c07909e1c4 not found: ID does not exist" containerID="52620863b40c6532af69c3596ca95bfd43c84a1a819923d6ab7942c07909e1c4" Jul 30 20:11:33.303047 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:33.302975 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52620863b40c6532af69c3596ca95bfd43c84a1a819923d6ab7942c07909e1c4"} err="failed to get container status \"52620863b40c6532af69c3596ca95bfd43c84a1a819923d6ab7942c07909e1c4\": rpc error: code = NotFound desc = could not find container \"52620863b40c6532af69c3596ca95bfd43c84a1a819923d6ab7942c07909e1c4\": container with ID starting with 52620863b40c6532af69c3596ca95bfd43c84a1a819923d6ab7942c07909e1c4 not found: ID does not exist" Jul 30 20:11:33.303047 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:33.302992 2556 scope.go:117] "RemoveContainer" containerID="1cd18a16b2dcb3f7e86dd07211979ae731455c9847313e2338a0ff2e6a8834d2" Jul 30 20:11:33.312500 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:33.312484 2556 scope.go:117] "RemoveContainer" containerID="1cd18a16b2dcb3f7e86dd07211979ae731455c9847313e2338a0ff2e6a8834d2" Jul 30 20:11:33.312764 ip-10-0-129-105 kubenswrapper[2556]: E0730 20:11:33.312746 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cd18a16b2dcb3f7e86dd07211979ae731455c9847313e2338a0ff2e6a8834d2\": container with ID starting with 1cd18a16b2dcb3f7e86dd07211979ae731455c9847313e2338a0ff2e6a8834d2 not found: ID does not exist" containerID="1cd18a16b2dcb3f7e86dd07211979ae731455c9847313e2338a0ff2e6a8834d2" Jul 30 20:11:33.312822 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:33.312769 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cd18a16b2dcb3f7e86dd07211979ae731455c9847313e2338a0ff2e6a8834d2"} err="failed to get container status \"1cd18a16b2dcb3f7e86dd07211979ae731455c9847313e2338a0ff2e6a8834d2\": rpc error: code = NotFound desc = could not find container \"1cd18a16b2dcb3f7e86dd07211979ae731455c9847313e2338a0ff2e6a8834d2\": container with ID starting with 1cd18a16b2dcb3f7e86dd07211979ae731455c9847313e2338a0ff2e6a8834d2 not found: ID does not exist" Jul 30 20:11:33.314986 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:33.314967 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-18459c-69574c7858-47dk6"] Jul 30 20:11:33.319490 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:33.319468 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-18459c-69574c7858-47dk6"] Jul 30 20:11:33.336466 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:33.336448 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-18459c-768bbfdb88-25j9q"] Jul 30 20:11:33.340462 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:33.340444 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-18459c-768bbfdb88-25j9q"] Jul 30 20:11:34.323599 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:34.323567 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aabcfdfd-ce62-4e66-a4fd-acd3cc64e7dd" path="/var/lib/kubelet/pods/aabcfdfd-ce62-4e66-a4fd-acd3cc64e7dd/volumes" Jul 30 20:11:34.323961 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:34.323909 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc727233-80e3-4b89-94f6-e7ffea06c05d" path="/var/lib/kubelet/pods/cc727233-80e3-4b89-94f6-e7ffea06c05d/volumes" Jul 30 20:11:38.587294 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:38.587266 2556 scope.go:117] "RemoveContainer" containerID="43bf3b36cfb3b90136bc334d7778e5da8981fa72a00118587413e7d43c114295" Jul 30 20:11:44.459999 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:44.459966 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-ecda00-6bdfb65bb8-kdmpc"] Jul 30 20:11:44.460410 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:44.460392 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d90b7e6c-2481-4909-8e10-33a7be2da209" containerName="payload-pre-processing" Jul 30 20:11:44.460464 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:44.460412 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d90b7e6c-2481-4909-8e10-33a7be2da209" containerName="payload-pre-processing" Jul 30 20:11:44.460464 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:44.460438 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44a0293f-d6e9-4c37-b2b0-7c77e0aaa54a" containerName="payload-processing" Jul 30 20:11:44.460464 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:44.460447 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="44a0293f-d6e9-4c37-b2b0-7c77e0aaa54a" containerName="payload-processing" Jul 30 20:11:44.460557 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:44.460472 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aabcfdfd-ce62-4e66-a4fd-acd3cc64e7dd" containerName="payload-pre-processing" Jul 30 20:11:44.460557 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:44.460478 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="aabcfdfd-ce62-4e66-a4fd-acd3cc64e7dd" containerName="payload-pre-processing" Jul 30 20:11:44.460557 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:44.460485 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc727233-80e3-4b89-94f6-e7ffea06c05d" containerName="payload-processing" Jul 30 20:11:44.460557 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:44.460490 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc727233-80e3-4b89-94f6-e7ffea06c05d" containerName="payload-processing" Jul 30 20:11:44.460557 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:44.460555 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc727233-80e3-4b89-94f6-e7ffea06c05d" containerName="payload-processing" Jul 30 20:11:44.460716 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:44.460566 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d90b7e6c-2481-4909-8e10-33a7be2da209" containerName="payload-pre-processing" Jul 30 20:11:44.460716 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:44.460573 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="aabcfdfd-ce62-4e66-a4fd-acd3cc64e7dd" containerName="payload-pre-processing" Jul 30 20:11:44.460716 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:44.460582 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="44a0293f-d6e9-4c37-b2b0-7c77e0aaa54a" containerName="payload-processing" Jul 30 20:11:44.465338 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:44.465321 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ecda00-6bdfb65bb8-kdmpc" Jul 30 20:11:44.468010 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:44.467984 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-ecda00\"" Jul 30 20:11:44.468157 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:44.468046 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-ecda00-dockercfg-v72mm\"" Jul 30 20:11:44.474670 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:44.474647 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-ecda00-6bdfb65bb8-kdmpc"] Jul 30 20:11:44.487323 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:44.487282 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-ecda00-6d5df66494-nkq2b"] Jul 30 20:11:44.491275 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:44.491259 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-ecda00-6d5df66494-nkq2b" Jul 30 20:11:44.497730 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:44.497709 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-ecda00-6d5df66494-nkq2b"] Jul 30 20:11:44.574295 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:44.574263 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d902f0c2-c297-4956-857e-63a11ffcae73-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-ecda00-6bdfb65bb8-kdmpc\" (UID: \"d902f0c2-c297-4956-857e-63a11ffcae73\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ecda00-6bdfb65bb8-kdmpc" Jul 30 20:11:44.574460 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:44.574310 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbnrs\" (UniqueName: \"kubernetes.io/projected/a6e6af4a-dc00-4746-b1d8-9a3541cc880a-kube-api-access-lbnrs\") pod \"payload-processing-e2e-shared-a-ecda00-6d5df66494-nkq2b\" (UID: \"a6e6af4a-dc00-4746-b1d8-9a3541cc880a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-ecda00-6d5df66494-nkq2b" Jul 30 20:11:44.574460 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:44.574371 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwq8g\" (UniqueName: \"kubernetes.io/projected/d902f0c2-c297-4956-857e-63a11ffcae73-kube-api-access-lwq8g\") pod \"payload-pre-processing-e2e-shared-a-ecda00-6bdfb65bb8-kdmpc\" (UID: \"d902f0c2-c297-4956-857e-63a11ffcae73\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ecda00-6bdfb65bb8-kdmpc" Jul 30 20:11:44.574460 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:44.574455 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6e6af4a-dc00-4746-b1d8-9a3541cc880a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-ecda00-6d5df66494-nkq2b\" (UID: \"a6e6af4a-dc00-4746-b1d8-9a3541cc880a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-ecda00-6d5df66494-nkq2b" Jul 30 20:11:44.675304 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:44.675267 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d902f0c2-c297-4956-857e-63a11ffcae73-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-ecda00-6bdfb65bb8-kdmpc\" (UID: \"d902f0c2-c297-4956-857e-63a11ffcae73\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ecda00-6bdfb65bb8-kdmpc" Jul 30 20:11:44.675487 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:44.675318 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbnrs\" (UniqueName: \"kubernetes.io/projected/a6e6af4a-dc00-4746-b1d8-9a3541cc880a-kube-api-access-lbnrs\") pod \"payload-processing-e2e-shared-a-ecda00-6d5df66494-nkq2b\" (UID: \"a6e6af4a-dc00-4746-b1d8-9a3541cc880a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-ecda00-6d5df66494-nkq2b" Jul 30 20:11:44.675535 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:44.675513 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwq8g\" (UniqueName: \"kubernetes.io/projected/d902f0c2-c297-4956-857e-63a11ffcae73-kube-api-access-lwq8g\") pod \"payload-pre-processing-e2e-shared-a-ecda00-6bdfb65bb8-kdmpc\" (UID: \"d902f0c2-c297-4956-857e-63a11ffcae73\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ecda00-6bdfb65bb8-kdmpc" Jul 30 20:11:44.675569 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:44.675554 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6e6af4a-dc00-4746-b1d8-9a3541cc880a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-ecda00-6d5df66494-nkq2b\" (UID: \"a6e6af4a-dc00-4746-b1d8-9a3541cc880a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-ecda00-6d5df66494-nkq2b" Jul 30 20:11:44.675994 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:44.675975 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d902f0c2-c297-4956-857e-63a11ffcae73-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-ecda00-6bdfb65bb8-kdmpc\" (UID: \"d902f0c2-c297-4956-857e-63a11ffcae73\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ecda00-6bdfb65bb8-kdmpc" Jul 30 20:11:44.676239 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:44.676221 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6e6af4a-dc00-4746-b1d8-9a3541cc880a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-ecda00-6d5df66494-nkq2b\" (UID: \"a6e6af4a-dc00-4746-b1d8-9a3541cc880a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-ecda00-6d5df66494-nkq2b" Jul 30 20:11:44.683265 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:44.683232 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwq8g\" (UniqueName: \"kubernetes.io/projected/d902f0c2-c297-4956-857e-63a11ffcae73-kube-api-access-lwq8g\") pod \"payload-pre-processing-e2e-shared-a-ecda00-6bdfb65bb8-kdmpc\" (UID: \"d902f0c2-c297-4956-857e-63a11ffcae73\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ecda00-6bdfb65bb8-kdmpc" Jul 30 20:11:44.683410 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:44.683328 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbnrs\" (UniqueName: \"kubernetes.io/projected/a6e6af4a-dc00-4746-b1d8-9a3541cc880a-kube-api-access-lbnrs\") pod \"payload-processing-e2e-shared-a-ecda00-6d5df66494-nkq2b\" (UID: \"a6e6af4a-dc00-4746-b1d8-9a3541cc880a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-ecda00-6d5df66494-nkq2b" Jul 30 20:11:44.777106 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:44.777039 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ecda00-6bdfb65bb8-kdmpc" Jul 30 20:11:44.804016 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:44.803995 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-ecda00-6d5df66494-nkq2b" Jul 30 20:11:45.126602 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:45.126580 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-ecda00-6bdfb65bb8-kdmpc"] Jul 30 20:11:45.128491 ip-10-0-129-105 kubenswrapper[2556]: W0730 20:11:45.128465 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd902f0c2_c297_4956_857e_63a11ffcae73.slice/crio-ef129ebfa67201360cf001a1a977572f29f0d2b3967bfe04a5daadfc4bcdd6cc WatchSource:0}: Error finding container ef129ebfa67201360cf001a1a977572f29f0d2b3967bfe04a5daadfc4bcdd6cc: Status 404 returned error can't find the container with id ef129ebfa67201360cf001a1a977572f29f0d2b3967bfe04a5daadfc4bcdd6cc Jul 30 20:11:45.156833 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:45.156815 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-ecda00-6d5df66494-nkq2b"] Jul 30 20:11:45.162405 ip-10-0-129-105 kubenswrapper[2556]: W0730 20:11:45.162367 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6e6af4a_dc00_4746_b1d8_9a3541cc880a.slice/crio-75283bf495cb226b4347da25649b21ced3cc88de7bda94cbb22f753f6776c90a WatchSource:0}: Error finding container 75283bf495cb226b4347da25649b21ced3cc88de7bda94cbb22f753f6776c90a: Status 404 returned error can't find the container with id 75283bf495cb226b4347da25649b21ced3cc88de7bda94cbb22f753f6776c90a Jul 30 20:11:45.366873 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:45.366834 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-ecda00-6d5df66494-nkq2b" event={"ID":"a6e6af4a-dc00-4746-b1d8-9a3541cc880a","Type":"ContainerStarted","Data":"28100c8976e96f8ddee2927cbb642ebcb8de6dbbd8f52ba6f12e0260b135a4a6"} Jul 30 20:11:45.367039 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:45.366932 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-ecda00-6d5df66494-nkq2b" event={"ID":"a6e6af4a-dc00-4746-b1d8-9a3541cc880a","Type":"ContainerStarted","Data":"75283bf495cb226b4347da25649b21ced3cc88de7bda94cbb22f753f6776c90a"} Jul 30 20:11:45.367039 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:45.366964 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-ecda00-6d5df66494-nkq2b" Jul 30 20:11:45.368359 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:45.368334 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ecda00-6bdfb65bb8-kdmpc" event={"ID":"d902f0c2-c297-4956-857e-63a11ffcae73","Type":"ContainerStarted","Data":"1d5bb43118cdf0f1732bc77f78337e6a07b5de176491b3748d19748c8302f5b5"} Jul 30 20:11:45.368449 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:45.368377 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ecda00-6bdfb65bb8-kdmpc" event={"ID":"d902f0c2-c297-4956-857e-63a11ffcae73","Type":"ContainerStarted","Data":"ef129ebfa67201360cf001a1a977572f29f0d2b3967bfe04a5daadfc4bcdd6cc"} Jul 30 20:11:45.368449 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:45.368413 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ecda00-6bdfb65bb8-kdmpc" Jul 30 20:11:45.383323 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:45.383245 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-ecda00-6d5df66494-nkq2b" podStartSLOduration=1.383232194 podStartE2EDuration="1.383232194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:11:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:11:45.381773379 +0000 UTC m=+2827.601332294" watchObservedRunningTime="2026-07-30 20:11:45.383232194 +0000 UTC m=+2827.602791124" Jul 30 20:11:45.397060 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:45.397024 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ecda00-6bdfb65bb8-kdmpc" podStartSLOduration=1.397010907 podStartE2EDuration="1.397010907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:11:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:11:45.39564441 +0000 UTC m=+2827.615203324" watchObservedRunningTime="2026-07-30 20:11:45.397010907 +0000 UTC m=+2827.616569819" Jul 30 20:11:56.381780 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:56.381750 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ecda00-6bdfb65bb8-kdmpc" Jul 30 20:11:56.382261 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:11:56.382039 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-ecda00-6d5df66494-nkq2b" Jul 30 20:12:13.135514 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:12:13.135478 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-a769e5-5c697f9788-vmjhv"] Jul 30 20:12:13.139613 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:12:13.139590 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-a769e5-5c697f9788-vmjhv" Jul 30 20:12:13.141867 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:12:13.141849 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-a769e5-dockercfg-rgg6b\"" Jul 30 20:12:13.141979 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:12:13.141959 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-a769e5\"" Jul 30 20:12:13.148929 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:12:13.148910 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-a769e5-5c697f9788-vmjhv"] Jul 30 20:12:13.175788 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:12:13.175768 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-a769e5-6fd4dbbf9d-wzpq9"] Jul 30 20:12:13.179322 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:12:13.179306 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-a769e5-6fd4dbbf9d-wzpq9" Jul 30 20:12:13.190216 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:12:13.190195 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-a769e5-6fd4dbbf9d-wzpq9"] Jul 30 20:12:13.232634 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:12:13.232603 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f759bb7-6a34-4043-bf5f-78d69c7b305f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-a769e5-5c697f9788-vmjhv\" (UID: \"7f759bb7-6a34-4043-bf5f-78d69c7b305f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-a769e5-5c697f9788-vmjhv" Jul 30 20:12:13.232755 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:12:13.232735 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/645b7fef-fa10-4957-91af-a5c1fef25806-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-a769e5-6fd4dbbf9d-wzpq9\" (UID: \"645b7fef-fa10-4957-91af-a5c1fef25806\") " pod="openshift-ingress/payload-processing-e2e-shared-b-a769e5-6fd4dbbf9d-wzpq9" Jul 30 20:12:13.232828 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:12:13.232806 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6jmg\" (UniqueName: \"kubernetes.io/projected/645b7fef-fa10-4957-91af-a5c1fef25806-kube-api-access-s6jmg\") pod \"payload-processing-e2e-shared-b-a769e5-6fd4dbbf9d-wzpq9\" (UID: \"645b7fef-fa10-4957-91af-a5c1fef25806\") " pod="openshift-ingress/payload-processing-e2e-shared-b-a769e5-6fd4dbbf9d-wzpq9" Jul 30 20:12:13.232929 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:12:13.232914 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htgf9\" (UniqueName: \"kubernetes.io/projected/7f759bb7-6a34-4043-bf5f-78d69c7b305f-kube-api-access-htgf9\") pod \"payload-pre-processing-e2e-shared-b-a769e5-5c697f9788-vmjhv\" (UID: \"7f759bb7-6a34-4043-bf5f-78d69c7b305f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-a769e5-5c697f9788-vmjhv" Jul 30 20:12:13.333738 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:12:13.333702 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6jmg\" (UniqueName: \"kubernetes.io/projected/645b7fef-fa10-4957-91af-a5c1fef25806-kube-api-access-s6jmg\") pod \"payload-processing-e2e-shared-b-a769e5-6fd4dbbf9d-wzpq9\" (UID: \"645b7fef-fa10-4957-91af-a5c1fef25806\") " pod="openshift-ingress/payload-processing-e2e-shared-b-a769e5-6fd4dbbf9d-wzpq9" Jul 30 20:12:13.333893 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:12:13.333782 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htgf9\" (UniqueName: \"kubernetes.io/projected/7f759bb7-6a34-4043-bf5f-78d69c7b305f-kube-api-access-htgf9\") pod \"payload-pre-processing-e2e-shared-b-a769e5-5c697f9788-vmjhv\" (UID: \"7f759bb7-6a34-4043-bf5f-78d69c7b305f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-a769e5-5c697f9788-vmjhv" Jul 30 20:12:13.333893 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:12:13.333831 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f759bb7-6a34-4043-bf5f-78d69c7b305f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-a769e5-5c697f9788-vmjhv\" (UID: \"7f759bb7-6a34-4043-bf5f-78d69c7b305f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-a769e5-5c697f9788-vmjhv" Jul 30 20:12:13.333973 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:12:13.333910 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/645b7fef-fa10-4957-91af-a5c1fef25806-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-a769e5-6fd4dbbf9d-wzpq9\" (UID: \"645b7fef-fa10-4957-91af-a5c1fef25806\") " pod="openshift-ingress/payload-processing-e2e-shared-b-a769e5-6fd4dbbf9d-wzpq9" Jul 30 20:12:13.334609 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:12:13.334591 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/645b7fef-fa10-4957-91af-a5c1fef25806-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-a769e5-6fd4dbbf9d-wzpq9\" (UID: \"645b7fef-fa10-4957-91af-a5c1fef25806\") " pod="openshift-ingress/payload-processing-e2e-shared-b-a769e5-6fd4dbbf9d-wzpq9" Jul 30 20:12:13.334658 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:12:13.334610 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f759bb7-6a34-4043-bf5f-78d69c7b305f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-a769e5-5c697f9788-vmjhv\" (UID: \"7f759bb7-6a34-4043-bf5f-78d69c7b305f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-a769e5-5c697f9788-vmjhv" Jul 30 20:12:13.341178 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:12:13.341152 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htgf9\" (UniqueName: \"kubernetes.io/projected/7f759bb7-6a34-4043-bf5f-78d69c7b305f-kube-api-access-htgf9\") pod \"payload-pre-processing-e2e-shared-b-a769e5-5c697f9788-vmjhv\" (UID: \"7f759bb7-6a34-4043-bf5f-78d69c7b305f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-a769e5-5c697f9788-vmjhv" Jul 30 20:12:13.341374 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:12:13.341352 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6jmg\" (UniqueName: \"kubernetes.io/projected/645b7fef-fa10-4957-91af-a5c1fef25806-kube-api-access-s6jmg\") pod \"payload-processing-e2e-shared-b-a769e5-6fd4dbbf9d-wzpq9\" (UID: \"645b7fef-fa10-4957-91af-a5c1fef25806\") " pod="openshift-ingress/payload-processing-e2e-shared-b-a769e5-6fd4dbbf9d-wzpq9" Jul 30 20:12:13.452202 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:12:13.452121 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-a769e5-5c697f9788-vmjhv" Jul 30 20:12:13.490099 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:12:13.490070 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-a769e5-6fd4dbbf9d-wzpq9" Jul 30 20:12:13.639368 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:12:13.639341 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-a769e5-6fd4dbbf9d-wzpq9"] Jul 30 20:12:13.641373 ip-10-0-129-105 kubenswrapper[2556]: W0730 20:12:13.641343 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod645b7fef_fa10_4957_91af_a5c1fef25806.slice/crio-ef343dd23a185ce985065ff7a318b9fadf8b2a0933f0173a1e0cd77f93137b33 WatchSource:0}: Error finding container ef343dd23a185ce985065ff7a318b9fadf8b2a0933f0173a1e0cd77f93137b33: Status 404 returned error can't find the container with id ef343dd23a185ce985065ff7a318b9fadf8b2a0933f0173a1e0cd77f93137b33 Jul 30 20:12:13.803443 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:12:13.803414 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-a769e5-5c697f9788-vmjhv"] Jul 30 20:12:13.804747 ip-10-0-129-105 kubenswrapper[2556]: W0730 20:12:13.804722 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f759bb7_6a34_4043_bf5f_78d69c7b305f.slice/crio-b56ff0e8e83fb7681173de3f3804e0237ffe3047c3ade8d90145577d09bb6420 WatchSource:0}: Error finding container b56ff0e8e83fb7681173de3f3804e0237ffe3047c3ade8d90145577d09bb6420: Status 404 returned error can't find the container with id b56ff0e8e83fb7681173de3f3804e0237ffe3047c3ade8d90145577d09bb6420 Jul 30 20:12:14.530160 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:12:14.530130 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-a769e5-5c697f9788-vmjhv" event={"ID":"7f759bb7-6a34-4043-bf5f-78d69c7b305f","Type":"ContainerStarted","Data":"cf0c95638853747da933a784bff4c613a9663bdba3079894179d7120d9ebfc52"} Jul 30 20:12:14.530641 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:12:14.530193 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-a769e5-5c697f9788-vmjhv" Jul 30 20:12:14.530641 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:12:14.530218 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-a769e5-5c697f9788-vmjhv" event={"ID":"7f759bb7-6a34-4043-bf5f-78d69c7b305f","Type":"ContainerStarted","Data":"b56ff0e8e83fb7681173de3f3804e0237ffe3047c3ade8d90145577d09bb6420"} Jul 30 20:12:14.531544 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:12:14.531516 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-a769e5-6fd4dbbf9d-wzpq9" event={"ID":"645b7fef-fa10-4957-91af-a5c1fef25806","Type":"ContainerStarted","Data":"6e132fc8d263f68c90b558f7e09d15acdc389f426ba515726314abf966f96f64"} Jul 30 20:12:14.531634 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:12:14.531567 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-a769e5-6fd4dbbf9d-wzpq9" event={"ID":"645b7fef-fa10-4957-91af-a5c1fef25806","Type":"ContainerStarted","Data":"ef343dd23a185ce985065ff7a318b9fadf8b2a0933f0173a1e0cd77f93137b33"} Jul 30 20:12:14.531634 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:12:14.531599 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-a769e5-6fd4dbbf9d-wzpq9" Jul 30 20:12:14.545837 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:12:14.545788 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-a769e5-5c697f9788-vmjhv" podStartSLOduration=1.545775553 podStartE2EDuration="1.545775553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:12:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:12:14.544567066 +0000 UTC m=+2856.764125985" watchObservedRunningTime="2026-07-30 20:12:14.545775553 +0000 UTC m=+2856.765334466" Jul 30 20:12:14.562001 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:12:14.561961 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-a769e5-6fd4dbbf9d-wzpq9" podStartSLOduration=1.561947726 podStartE2EDuration="1.561947726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:12:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:12:14.559553042 +0000 UTC m=+2856.779111955" watchObservedRunningTime="2026-07-30 20:12:14.561947726 +0000 UTC m=+2856.781506638" Jul 30 20:12:25.547168 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:12:25.547138 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-a769e5-6fd4dbbf9d-wzpq9" Jul 30 20:12:25.547673 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:12:25.547213 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-a769e5-5c697f9788-vmjhv" Jul 30 20:14:38.474247 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:14:38.474222 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-795qw_4cdd383f-6493-4d83-80fa-f4ce17e8416e/ovn-acl-logging/0.log" Jul 30 20:14:38.487711 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:14:38.487687 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-795qw_4cdd383f-6493-4d83-80fa-f4ce17e8416e/ovn-acl-logging/0.log" Jul 30 20:19:38.513459 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:19:38.513427 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-795qw_4cdd383f-6493-4d83-80fa-f4ce17e8416e/ovn-acl-logging/0.log" Jul 30 20:19:38.522369 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:19:38.522350 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-795qw_4cdd383f-6493-4d83-80fa-f4ce17e8416e/ovn-acl-logging/0.log" Jul 30 20:23:29.819576 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:23:29.819545 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0c599c-66f5b555bd-f4cd2"] Jul 30 20:23:29.823203 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:23:29.823187 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c599c-66f5b555bd-f4cd2" Jul 30 20:23:29.826370 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:23:29.826348 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-0c599c-dockercfg-k79xn\"" Jul 30 20:23:29.826370 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:23:29.826348 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-0c599c\"" Jul 30 20:23:29.833432 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:23:29.833406 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0c599c-66f5b555bd-f4cd2"] Jul 30 20:23:29.856694 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:23:29.856665 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0c599c-c649cc66d-r5zjv"] Jul 30 20:23:29.859507 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:23:29.859482 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcn89\" (UniqueName: \"kubernetes.io/projected/94b47a93-6faa-43fd-a35d-d951180afba2-kube-api-access-lcn89\") pod \"payload-pre-processing-e2e-ipp-a-0c599c-66f5b555bd-f4cd2\" (UID: \"94b47a93-6faa-43fd-a35d-d951180afba2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c599c-66f5b555bd-f4cd2" Jul 30 20:23:29.859602 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:23:29.859517 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94b47a93-6faa-43fd-a35d-d951180afba2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0c599c-66f5b555bd-f4cd2\" (UID: \"94b47a93-6faa-43fd-a35d-d951180afba2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c599c-66f5b555bd-f4cd2" Jul 30 20:23:29.860608 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:23:29.860594 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0c599c-c649cc66d-r5zjv" Jul 30 20:23:29.870119 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:23:29.870099 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0c599c-c649cc66d-r5zjv"] Jul 30 20:23:29.960917 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:23:29.960876 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcn89\" (UniqueName: \"kubernetes.io/projected/94b47a93-6faa-43fd-a35d-d951180afba2-kube-api-access-lcn89\") pod \"payload-pre-processing-e2e-ipp-a-0c599c-66f5b555bd-f4cd2\" (UID: \"94b47a93-6faa-43fd-a35d-d951180afba2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c599c-66f5b555bd-f4cd2" Jul 30 20:23:29.960917 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:23:29.960917 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94b47a93-6faa-43fd-a35d-d951180afba2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0c599c-66f5b555bd-f4cd2\" (UID: \"94b47a93-6faa-43fd-a35d-d951180afba2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c599c-66f5b555bd-f4cd2" Jul 30 20:23:29.961163 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:23:29.960952 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32caabdc-5d4c-4a3b-965b-edc180546f1a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0c599c-c649cc66d-r5zjv\" (UID: \"32caabdc-5d4c-4a3b-965b-edc180546f1a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0c599c-c649cc66d-r5zjv" Jul 30 20:23:29.961228 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:23:29.961163 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bm7x\" (UniqueName: \"kubernetes.io/projected/32caabdc-5d4c-4a3b-965b-edc180546f1a-kube-api-access-8bm7x\") pod \"payload-processing-e2e-ipp-a-0c599c-c649cc66d-r5zjv\" (UID: \"32caabdc-5d4c-4a3b-965b-edc180546f1a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0c599c-c649cc66d-r5zjv" Jul 30 20:23:29.961710 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:23:29.961693 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94b47a93-6faa-43fd-a35d-d951180afba2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0c599c-66f5b555bd-f4cd2\" (UID: \"94b47a93-6faa-43fd-a35d-d951180afba2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c599c-66f5b555bd-f4cd2" Jul 30 20:23:29.968306 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:23:29.968287 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcn89\" (UniqueName: \"kubernetes.io/projected/94b47a93-6faa-43fd-a35d-d951180afba2-kube-api-access-lcn89\") pod \"payload-pre-processing-e2e-ipp-a-0c599c-66f5b555bd-f4cd2\" (UID: \"94b47a93-6faa-43fd-a35d-d951180afba2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c599c-66f5b555bd-f4cd2" Jul 30 20:23:30.062420 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:23:30.062371 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bm7x\" (UniqueName: \"kubernetes.io/projected/32caabdc-5d4c-4a3b-965b-edc180546f1a-kube-api-access-8bm7x\") pod \"payload-processing-e2e-ipp-a-0c599c-c649cc66d-r5zjv\" (UID: \"32caabdc-5d4c-4a3b-965b-edc180546f1a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0c599c-c649cc66d-r5zjv" Jul 30 20:23:30.062556 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:23:30.062445 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32caabdc-5d4c-4a3b-965b-edc180546f1a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0c599c-c649cc66d-r5zjv\" (UID: \"32caabdc-5d4c-4a3b-965b-edc180546f1a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0c599c-c649cc66d-r5zjv" Jul 30 20:23:30.063063 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:23:30.063047 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32caabdc-5d4c-4a3b-965b-edc180546f1a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0c599c-c649cc66d-r5zjv\" (UID: \"32caabdc-5d4c-4a3b-965b-edc180546f1a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0c599c-c649cc66d-r5zjv" Jul 30 20:23:30.069886 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:23:30.069825 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bm7x\" (UniqueName: \"kubernetes.io/projected/32caabdc-5d4c-4a3b-965b-edc180546f1a-kube-api-access-8bm7x\") pod \"payload-processing-e2e-ipp-a-0c599c-c649cc66d-r5zjv\" (UID: \"32caabdc-5d4c-4a3b-965b-edc180546f1a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0c599c-c649cc66d-r5zjv" Jul 30 20:23:30.134885 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:23:30.134852 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c599c-66f5b555bd-f4cd2" Jul 30 20:23:30.171859 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:23:30.171817 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0c599c-c649cc66d-r5zjv" Jul 30 20:23:30.495802 ip-10-0-129-105 kubenswrapper[2556]: W0730 20:23:30.495767 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94b47a93_6faa_43fd_a35d_d951180afba2.slice/crio-01ab1f9d43f73af74f8112056d2c209baeca8306e745d9effe1969ef543cd801 WatchSource:0}: Error finding container 01ab1f9d43f73af74f8112056d2c209baeca8306e745d9effe1969ef543cd801: Status 404 returned error can't find the container with id 01ab1f9d43f73af74f8112056d2c209baeca8306e745d9effe1969ef543cd801 Jul 30 20:23:30.499204 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:23:30.499188 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0c599c-66f5b555bd-f4cd2"] Jul 30 20:23:30.534957 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:23:30.534936 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0c599c-c649cc66d-r5zjv"] Jul 30 20:23:30.538538 ip-10-0-129-105 kubenswrapper[2556]: W0730 20:23:30.538512 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32caabdc_5d4c_4a3b_965b_edc180546f1a.slice/crio-0ef4e2c79a4ac78592452c01722f62c524945bc82fdd9e789f321fc50add0d5d WatchSource:0}: Error finding container 0ef4e2c79a4ac78592452c01722f62c524945bc82fdd9e789f321fc50add0d5d: Status 404 returned error can't find the container with id 0ef4e2c79a4ac78592452c01722f62c524945bc82fdd9e789f321fc50add0d5d Jul 30 20:23:31.026506 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:23:31.026468 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c599c-66f5b555bd-f4cd2" event={"ID":"94b47a93-6faa-43fd-a35d-d951180afba2","Type":"ContainerStarted","Data":"6647f28f936c17031168fd25c6e22ad8eaf0c65026df4b11e24b49e3fdab5686"} Jul 30 20:23:31.026951 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:23:31.026527 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c599c-66f5b555bd-f4cd2" event={"ID":"94b47a93-6faa-43fd-a35d-d951180afba2","Type":"ContainerStarted","Data":"01ab1f9d43f73af74f8112056d2c209baeca8306e745d9effe1969ef543cd801"} Jul 30 20:23:31.026951 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:23:31.026558 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c599c-66f5b555bd-f4cd2" Jul 30 20:23:31.028107 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:23:31.028083 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0c599c-c649cc66d-r5zjv" event={"ID":"32caabdc-5d4c-4a3b-965b-edc180546f1a","Type":"ContainerStarted","Data":"b4066c05082096ff7c2488deb36cccd3c5562afab6d9f3c5371be7dd40e70954"} Jul 30 20:23:31.028223 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:23:31.028126 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0c599c-c649cc66d-r5zjv" event={"ID":"32caabdc-5d4c-4a3b-965b-edc180546f1a","Type":"ContainerStarted","Data":"0ef4e2c79a4ac78592452c01722f62c524945bc82fdd9e789f321fc50add0d5d"} Jul 30 20:23:31.028223 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:23:31.028145 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0c599c-c649cc66d-r5zjv" Jul 30 20:23:31.045795 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:23:31.045750 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c599c-66f5b555bd-f4cd2" podStartSLOduration=2.045736098 podStartE2EDuration="2.045736098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:23:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:23:31.04263721 +0000 UTC m=+3533.262196128" watchObservedRunningTime="2026-07-30 20:23:31.045736098 +0000 UTC m=+3533.265295010" Jul 30 20:23:42.036044 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:23:42.036008 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c599c-66f5b555bd-f4cd2" Jul 30 20:23:42.036496 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:23:42.036146 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0c599c-c649cc66d-r5zjv" Jul 30 20:23:42.053740 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:23:42.053687 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-0c599c-c649cc66d-r5zjv" podStartSLOduration=13.053674514 podStartE2EDuration="13.053674514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:23:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:23:31.05892004 +0000 UTC m=+3533.278478955" watchObservedRunningTime="2026-07-30 20:23:42.053674514 +0000 UTC m=+3544.273233426" Jul 30 20:23:59.005754 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:23:59.005720 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d67d52-58cc76fb6d-wmg7n"] Jul 30 20:23:59.009479 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:23:59.009457 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d67d52-58cc76fb6d-wmg7n" Jul 30 20:23:59.011871 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:23:59.011851 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d67d52\"" Jul 30 20:23:59.012003 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:23:59.011919 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-d67d52-dockercfg-76wx5\"" Jul 30 20:23:59.019625 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:23:59.019597 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d67d52-58cc76fb6d-wmg7n"] Jul 30 20:23:59.132611 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:23:59.132570 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0447a80f-534f-4c9b-ae4b-b077c7882b28-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d67d52-58cc76fb6d-wmg7n\" (UID: \"0447a80f-534f-4c9b-ae4b-b077c7882b28\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d67d52-58cc76fb6d-wmg7n" Jul 30 20:23:59.132779 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:23:59.132674 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64zdc\" (UniqueName: \"kubernetes.io/projected/0447a80f-534f-4c9b-ae4b-b077c7882b28-kube-api-access-64zdc\") pod \"payload-pre-processing-e2e-ipp-b-d67d52-58cc76fb6d-wmg7n\" (UID: \"0447a80f-534f-4c9b-ae4b-b077c7882b28\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d67d52-58cc76fb6d-wmg7n" Jul 30 20:23:59.233180 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:23:59.233143 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0447a80f-534f-4c9b-ae4b-b077c7882b28-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d67d52-58cc76fb6d-wmg7n\" (UID: \"0447a80f-534f-4c9b-ae4b-b077c7882b28\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d67d52-58cc76fb6d-wmg7n" Jul 30 20:23:59.233360 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:23:59.233204 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64zdc\" (UniqueName: \"kubernetes.io/projected/0447a80f-534f-4c9b-ae4b-b077c7882b28-kube-api-access-64zdc\") pod \"payload-pre-processing-e2e-ipp-b-d67d52-58cc76fb6d-wmg7n\" (UID: \"0447a80f-534f-4c9b-ae4b-b077c7882b28\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d67d52-58cc76fb6d-wmg7n" Jul 30 20:23:59.233994 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:23:59.233972 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0447a80f-534f-4c9b-ae4b-b077c7882b28-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d67d52-58cc76fb6d-wmg7n\" (UID: \"0447a80f-534f-4c9b-ae4b-b077c7882b28\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d67d52-58cc76fb6d-wmg7n" Jul 30 20:23:59.240985 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:23:59.240961 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64zdc\" (UniqueName: \"kubernetes.io/projected/0447a80f-534f-4c9b-ae4b-b077c7882b28-kube-api-access-64zdc\") pod \"payload-pre-processing-e2e-ipp-b-d67d52-58cc76fb6d-wmg7n\" (UID: \"0447a80f-534f-4c9b-ae4b-b077c7882b28\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d67d52-58cc76fb6d-wmg7n" Jul 30 20:23:59.321849 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:23:59.321778 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d67d52-58cc76fb6d-wmg7n" Jul 30 20:23:59.457725 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:23:59.457558 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d67d52-58cc76fb6d-wmg7n"] Jul 30 20:23:59.460079 ip-10-0-129-105 kubenswrapper[2556]: W0730 20:23:59.460046 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0447a80f_534f_4c9b_ae4b_b077c7882b28.slice/crio-f02e5c56da13322b3bcbcd03bfad95d8a2418feacaa4ca821c540e35654b35e8 WatchSource:0}: Error finding container f02e5c56da13322b3bcbcd03bfad95d8a2418feacaa4ca821c540e35654b35e8: Status 404 returned error can't find the container with id f02e5c56da13322b3bcbcd03bfad95d8a2418feacaa4ca821c540e35654b35e8 Jul 30 20:24:00.183784 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:24:00.183742 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d67d52-58cc76fb6d-wmg7n" event={"ID":"0447a80f-534f-4c9b-ae4b-b077c7882b28","Type":"ContainerStarted","Data":"8dd76b47c5ca6adb5ad90b9b09cd592152a6bc1491e12403b3882ba5df4ea248"} Jul 30 20:24:00.184171 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:24:00.183810 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d67d52-58cc76fb6d-wmg7n" Jul 30 20:24:00.184171 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:24:00.183834 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d67d52-58cc76fb6d-wmg7n" event={"ID":"0447a80f-534f-4c9b-ae4b-b077c7882b28","Type":"ContainerStarted","Data":"f02e5c56da13322b3bcbcd03bfad95d8a2418feacaa4ca821c540e35654b35e8"} Jul 30 20:24:00.201929 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:24:00.201881 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d67d52-58cc76fb6d-wmg7n" podStartSLOduration=2.201866621 podStartE2EDuration="2.201866621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:23:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:24:00.199795038 +0000 UTC m=+3562.419353957" watchObservedRunningTime="2026-07-30 20:24:00.201866621 +0000 UTC m=+3562.421425533" Jul 30 20:24:11.191243 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:24:11.191211 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d67d52-58cc76fb6d-wmg7n" Jul 30 20:24:29.109899 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:24:29.109866 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-knk59_bea701bd-a892-44cf-bedf-cbb6bd64bb37/payload-processing/0.log" Jul 30 20:24:38.546232 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:24:38.546205 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-795qw_4cdd383f-6493-4d83-80fa-f4ce17e8416e/ovn-acl-logging/0.log" Jul 30 20:24:38.555085 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:24:38.555067 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-795qw_4cdd383f-6493-4d83-80fa-f4ce17e8416e/ovn-acl-logging/0.log" Jul 30 20:25:26.336133 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:25:26.336107 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-0c599c-c649cc66d-r5zjv_32caabdc-5d4c-4a3b-965b-edc180546f1a/payload-processing/0.log" Jul 30 20:25:26.456076 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:25:26.456044 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-knk59_bea701bd-a892-44cf-bedf-cbb6bd64bb37/payload-processing/0.log" Jul 30 20:25:31.531595 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:25:31.531557 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-23c617-5755f6dcd8-qzdn8"] Jul 30 20:25:31.538462 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:25:31.536831 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-23c617-5755f6dcd8-qzdn8" Jul 30 20:25:31.541438 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:25:31.541405 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-23c617\"" Jul 30 20:25:31.541628 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:25:31.541564 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-23c617-dockercfg-x4f6d\"" Jul 30 20:25:31.547995 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:25:31.547971 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-23c617-5755f6dcd8-qzdn8"] Jul 30 20:25:31.556534 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:25:31.556510 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-23c617-566ff8cd48-ls9bw"] Jul 30 20:25:31.560327 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:25:31.560310 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-23c617-566ff8cd48-ls9bw" Jul 30 20:25:31.574326 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:25:31.574306 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-23c617-566ff8cd48-ls9bw"] Jul 30 20:25:31.615820 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:25:31.615788 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddfc42b4-cc19-4774-b5c9-fe2a628ef239-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-23c617-566ff8cd48-ls9bw\" (UID: \"ddfc42b4-cc19-4774-b5c9-fe2a628ef239\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-23c617-566ff8cd48-ls9bw" Jul 30 20:25:31.615927 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:25:31.615862 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7c92472-b77d-4d3b-8173-81f739cefcd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-23c617-5755f6dcd8-qzdn8\" (UID: \"b7c92472-b77d-4d3b-8173-81f739cefcd0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-23c617-5755f6dcd8-qzdn8" Jul 30 20:25:31.615991 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:25:31.615924 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rswvl\" (UniqueName: \"kubernetes.io/projected/ddfc42b4-cc19-4774-b5c9-fe2a628ef239-kube-api-access-rswvl\") pod \"payload-processing-e2e-ipp-cleanup-23c617-566ff8cd48-ls9bw\" (UID: \"ddfc42b4-cc19-4774-b5c9-fe2a628ef239\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-23c617-566ff8cd48-ls9bw" Jul 30 20:25:31.616040 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:25:31.616024 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzxhz\" (UniqueName: \"kubernetes.io/projected/b7c92472-b77d-4d3b-8173-81f739cefcd0-kube-api-access-jzxhz\") pod \"payload-pre-processing-e2e-ipp-cleanup-23c617-5755f6dcd8-qzdn8\" (UID: \"b7c92472-b77d-4d3b-8173-81f739cefcd0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-23c617-5755f6dcd8-qzdn8" Jul 30 20:25:31.717394 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:25:31.717358 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddfc42b4-cc19-4774-b5c9-fe2a628ef239-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-23c617-566ff8cd48-ls9bw\" (UID: \"ddfc42b4-cc19-4774-b5c9-fe2a628ef239\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-23c617-566ff8cd48-ls9bw" Jul 30 20:25:31.717558 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:25:31.717441 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7c92472-b77d-4d3b-8173-81f739cefcd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-23c617-5755f6dcd8-qzdn8\" (UID: \"b7c92472-b77d-4d3b-8173-81f739cefcd0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-23c617-5755f6dcd8-qzdn8" Jul 30 20:25:31.717558 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:25:31.717494 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rswvl\" (UniqueName: \"kubernetes.io/projected/ddfc42b4-cc19-4774-b5c9-fe2a628ef239-kube-api-access-rswvl\") pod \"payload-processing-e2e-ipp-cleanup-23c617-566ff8cd48-ls9bw\" (UID: \"ddfc42b4-cc19-4774-b5c9-fe2a628ef239\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-23c617-566ff8cd48-ls9bw" Jul 30 20:25:31.717671 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:25:31.717574 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzxhz\" (UniqueName: \"kubernetes.io/projected/b7c92472-b77d-4d3b-8173-81f739cefcd0-kube-api-access-jzxhz\") pod \"payload-pre-processing-e2e-ipp-cleanup-23c617-5755f6dcd8-qzdn8\" (UID: \"b7c92472-b77d-4d3b-8173-81f739cefcd0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-23c617-5755f6dcd8-qzdn8" Jul 30 20:25:31.718258 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:25:31.718232 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7c92472-b77d-4d3b-8173-81f739cefcd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-23c617-5755f6dcd8-qzdn8\" (UID: \"b7c92472-b77d-4d3b-8173-81f739cefcd0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-23c617-5755f6dcd8-qzdn8" Jul 30 20:25:31.718319 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:25:31.718289 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddfc42b4-cc19-4774-b5c9-fe2a628ef239-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-23c617-566ff8cd48-ls9bw\" (UID: \"ddfc42b4-cc19-4774-b5c9-fe2a628ef239\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-23c617-566ff8cd48-ls9bw" Jul 30 20:25:31.726331 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:25:31.726306 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzxhz\" (UniqueName: \"kubernetes.io/projected/b7c92472-b77d-4d3b-8173-81f739cefcd0-kube-api-access-jzxhz\") pod \"payload-pre-processing-e2e-ipp-cleanup-23c617-5755f6dcd8-qzdn8\" (UID: \"b7c92472-b77d-4d3b-8173-81f739cefcd0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-23c617-5755f6dcd8-qzdn8" Jul 30 20:25:31.726477 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:25:31.726314 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rswvl\" (UniqueName: \"kubernetes.io/projected/ddfc42b4-cc19-4774-b5c9-fe2a628ef239-kube-api-access-rswvl\") pod \"payload-processing-e2e-ipp-cleanup-23c617-566ff8cd48-ls9bw\" (UID: \"ddfc42b4-cc19-4774-b5c9-fe2a628ef239\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-23c617-566ff8cd48-ls9bw" Jul 30 20:25:31.849355 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:25:31.849296 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-23c617-5755f6dcd8-qzdn8" Jul 30 20:25:31.870336 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:25:31.870311 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-23c617-566ff8cd48-ls9bw" Jul 30 20:25:32.002133 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:25:32.002022 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-23c617-5755f6dcd8-qzdn8"] Jul 30 20:25:32.008048 ip-10-0-129-105 kubenswrapper[2556]: W0730 20:25:32.007999 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7c92472_b77d_4d3b_8173_81f739cefcd0.slice/crio-550eb050320963c4896afe0af3b5170cfd82d278cfef33d9cb912f1a1fef4d9f WatchSource:0}: Error finding container 550eb050320963c4896afe0af3b5170cfd82d278cfef33d9cb912f1a1fef4d9f: Status 404 returned error can't find the container with id 550eb050320963c4896afe0af3b5170cfd82d278cfef33d9cb912f1a1fef4d9f Jul 30 20:25:32.241090 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:25:32.241067 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-23c617-566ff8cd48-ls9bw"] Jul 30 20:25:32.242565 ip-10-0-129-105 kubenswrapper[2556]: W0730 20:25:32.242537 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddfc42b4_cc19_4774_b5c9_fe2a628ef239.slice/crio-6e153e38b3163785ffd13da69a1b1c538af1f96d2e4f736437814551f89ebc48 WatchSource:0}: Error finding container 6e153e38b3163785ffd13da69a1b1c538af1f96d2e4f736437814551f89ebc48: Status 404 returned error can't find the container with id 6e153e38b3163785ffd13da69a1b1c538af1f96d2e4f736437814551f89ebc48 Jul 30 20:25:32.660035 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:25:32.660001 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-23c617-566ff8cd48-ls9bw" event={"ID":"ddfc42b4-cc19-4774-b5c9-fe2a628ef239","Type":"ContainerStarted","Data":"a448bae615e963ecdc7de03bc927a54e512a76d078a1f96c70cb75a7b2013ea5"} Jul 30 20:25:32.660506 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:25:32.660053 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-23c617-566ff8cd48-ls9bw" event={"ID":"ddfc42b4-cc19-4774-b5c9-fe2a628ef239","Type":"ContainerStarted","Data":"6e153e38b3163785ffd13da69a1b1c538af1f96d2e4f736437814551f89ebc48"} Jul 30 20:25:32.660506 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:25:32.660117 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-23c617-566ff8cd48-ls9bw" Jul 30 20:25:32.661633 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:25:32.661609 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-23c617-5755f6dcd8-qzdn8" event={"ID":"b7c92472-b77d-4d3b-8173-81f739cefcd0","Type":"ContainerStarted","Data":"0eb8520ad36974bb7abb53f7e96d4ad030296a895056b629dac9642412735bd6"} Jul 30 20:25:32.661747 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:25:32.661651 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-23c617-5755f6dcd8-qzdn8" event={"ID":"b7c92472-b77d-4d3b-8173-81f739cefcd0","Type":"ContainerStarted","Data":"550eb050320963c4896afe0af3b5170cfd82d278cfef33d9cb912f1a1fef4d9f"} Jul 30 20:25:32.661747 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:25:32.661673 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-23c617-5755f6dcd8-qzdn8" Jul 30 20:25:32.679295 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:25:32.679246 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-23c617-566ff8cd48-ls9bw" podStartSLOduration=1.679233753 podStartE2EDuration="1.679233753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:25:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:25:32.677722313 +0000 UTC m=+3654.897281232" watchObservedRunningTime="2026-07-30 20:25:32.679233753 +0000 UTC m=+3654.898792695" Jul 30 20:25:32.693243 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:25:32.693206 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-23c617-5755f6dcd8-qzdn8" podStartSLOduration=1.693195019 podStartE2EDuration="1.693195019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:25:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:25:32.692735375 +0000 UTC m=+3654.912294309" watchObservedRunningTime="2026-07-30 20:25:32.693195019 +0000 UTC m=+3654.912753929" Jul 30 20:25:43.668898 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:25:43.668865 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-23c617-5755f6dcd8-qzdn8" Jul 30 20:25:43.669272 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:25:43.669111 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-23c617-566ff8cd48-ls9bw" Jul 30 20:26:01.450218 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:01.450117 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-23c617-566ff8cd48-ls9bw"] Jul 30 20:26:01.452559 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:01.450466 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-23c617-566ff8cd48-ls9bw" podUID="ddfc42b4-cc19-4774-b5c9-fe2a628ef239" containerName="payload-processing" containerID="cri-o://a448bae615e963ecdc7de03bc927a54e512a76d078a1f96c70cb75a7b2013ea5" gracePeriod=30 Jul 30 20:26:01.453048 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:01.452995 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-23c617-5755f6dcd8-qzdn8"] Jul 30 20:26:01.453256 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:01.453209 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-23c617-5755f6dcd8-qzdn8" podUID="b7c92472-b77d-4d3b-8173-81f739cefcd0" containerName="payload-pre-processing" containerID="cri-o://0eb8520ad36974bb7abb53f7e96d4ad030296a895056b629dac9642412735bd6" gracePeriod=30 Jul 30 20:26:01.719692 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:01.719665 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-23c617-5755f6dcd8-qzdn8" Jul 30 20:26:01.743423 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:01.743396 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-23c617-566ff8cd48-ls9bw" Jul 30 20:26:01.785139 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:01.785109 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzxhz\" (UniqueName: \"kubernetes.io/projected/b7c92472-b77d-4d3b-8173-81f739cefcd0-kube-api-access-jzxhz\") pod \"b7c92472-b77d-4d3b-8173-81f739cefcd0\" (UID: \"b7c92472-b77d-4d3b-8173-81f739cefcd0\") " Jul 30 20:26:01.785268 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:01.785168 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rswvl\" (UniqueName: \"kubernetes.io/projected/ddfc42b4-cc19-4774-b5c9-fe2a628ef239-kube-api-access-rswvl\") pod \"ddfc42b4-cc19-4774-b5c9-fe2a628ef239\" (UID: \"ddfc42b4-cc19-4774-b5c9-fe2a628ef239\") " Jul 30 20:26:01.785268 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:01.785192 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddfc42b4-cc19-4774-b5c9-fe2a628ef239-plugins-config-volume\") pod \"ddfc42b4-cc19-4774-b5c9-fe2a628ef239\" (UID: \"ddfc42b4-cc19-4774-b5c9-fe2a628ef239\") " Jul 30 20:26:01.785268 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:01.785249 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7c92472-b77d-4d3b-8173-81f739cefcd0-plugins-config-volume\") pod \"b7c92472-b77d-4d3b-8173-81f739cefcd0\" (UID: \"b7c92472-b77d-4d3b-8173-81f739cefcd0\") " Jul 30 20:26:01.785656 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:01.785628 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddfc42b4-cc19-4774-b5c9-fe2a628ef239-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ddfc42b4-cc19-4774-b5c9-fe2a628ef239" (UID: "ddfc42b4-cc19-4774-b5c9-fe2a628ef239"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 20:26:01.785760 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:01.785631 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7c92472-b77d-4d3b-8173-81f739cefcd0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b7c92472-b77d-4d3b-8173-81f739cefcd0" (UID: "b7c92472-b77d-4d3b-8173-81f739cefcd0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 20:26:01.787172 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:01.787153 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddfc42b4-cc19-4774-b5c9-fe2a628ef239-kube-api-access-rswvl" (OuterVolumeSpecName: "kube-api-access-rswvl") pod "ddfc42b4-cc19-4774-b5c9-fe2a628ef239" (UID: "ddfc42b4-cc19-4774-b5c9-fe2a628ef239"). InnerVolumeSpecName "kube-api-access-rswvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:26:01.787235 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:01.787189 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7c92472-b77d-4d3b-8173-81f739cefcd0-kube-api-access-jzxhz" (OuterVolumeSpecName: "kube-api-access-jzxhz") pod "b7c92472-b77d-4d3b-8173-81f739cefcd0" (UID: "b7c92472-b77d-4d3b-8173-81f739cefcd0"). InnerVolumeSpecName "kube-api-access-jzxhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:26:01.812511 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:01.812482 2556 generic.go:356] "Generic (PLEG): container finished" podID="ddfc42b4-cc19-4774-b5c9-fe2a628ef239" containerID="a448bae615e963ecdc7de03bc927a54e512a76d078a1f96c70cb75a7b2013ea5" exitCode=0 Jul 30 20:26:01.812622 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:01.812524 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-23c617-566ff8cd48-ls9bw" Jul 30 20:26:01.812622 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:01.812576 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-23c617-566ff8cd48-ls9bw" event={"ID":"ddfc42b4-cc19-4774-b5c9-fe2a628ef239","Type":"ContainerDied","Data":"a448bae615e963ecdc7de03bc927a54e512a76d078a1f96c70cb75a7b2013ea5"} Jul 30 20:26:01.812723 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:01.812634 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-23c617-566ff8cd48-ls9bw" event={"ID":"ddfc42b4-cc19-4774-b5c9-fe2a628ef239","Type":"ContainerDied","Data":"6e153e38b3163785ffd13da69a1b1c538af1f96d2e4f736437814551f89ebc48"} Jul 30 20:26:01.812723 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:01.812664 2556 scope.go:117] "RemoveContainer" containerID="a448bae615e963ecdc7de03bc927a54e512a76d078a1f96c70cb75a7b2013ea5" Jul 30 20:26:01.813991 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:01.813967 2556 generic.go:356] "Generic (PLEG): container finished" podID="b7c92472-b77d-4d3b-8173-81f739cefcd0" containerID="0eb8520ad36974bb7abb53f7e96d4ad030296a895056b629dac9642412735bd6" exitCode=0 Jul 30 20:26:01.814098 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:01.814008 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-23c617-5755f6dcd8-qzdn8" event={"ID":"b7c92472-b77d-4d3b-8173-81f739cefcd0","Type":"ContainerDied","Data":"0eb8520ad36974bb7abb53f7e96d4ad030296a895056b629dac9642412735bd6"} Jul 30 20:26:01.814098 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:01.814048 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-23c617-5755f6dcd8-qzdn8" event={"ID":"b7c92472-b77d-4d3b-8173-81f739cefcd0","Type":"ContainerDied","Data":"550eb050320963c4896afe0af3b5170cfd82d278cfef33d9cb912f1a1fef4d9f"} Jul 30 20:26:01.814098 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:01.814012 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-23c617-5755f6dcd8-qzdn8" Jul 30 20:26:01.825423 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:01.825406 2556 scope.go:117] "RemoveContainer" containerID="a448bae615e963ecdc7de03bc927a54e512a76d078a1f96c70cb75a7b2013ea5" Jul 30 20:26:01.825677 ip-10-0-129-105 kubenswrapper[2556]: E0730 20:26:01.825656 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a448bae615e963ecdc7de03bc927a54e512a76d078a1f96c70cb75a7b2013ea5\": container with ID starting with a448bae615e963ecdc7de03bc927a54e512a76d078a1f96c70cb75a7b2013ea5 not found: ID does not exist" containerID="a448bae615e963ecdc7de03bc927a54e512a76d078a1f96c70cb75a7b2013ea5" Jul 30 20:26:01.825741 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:01.825687 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a448bae615e963ecdc7de03bc927a54e512a76d078a1f96c70cb75a7b2013ea5"} err="failed to get container status \"a448bae615e963ecdc7de03bc927a54e512a76d078a1f96c70cb75a7b2013ea5\": rpc error: code = NotFound desc = could not find container \"a448bae615e963ecdc7de03bc927a54e512a76d078a1f96c70cb75a7b2013ea5\": container with ID starting with a448bae615e963ecdc7de03bc927a54e512a76d078a1f96c70cb75a7b2013ea5 not found: ID does not exist" Jul 30 20:26:01.825741 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:01.825704 2556 scope.go:117] "RemoveContainer" containerID="0eb8520ad36974bb7abb53f7e96d4ad030296a895056b629dac9642412735bd6" Jul 30 20:26:01.833520 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:01.833502 2556 scope.go:117] "RemoveContainer" containerID="0eb8520ad36974bb7abb53f7e96d4ad030296a895056b629dac9642412735bd6" Jul 30 20:26:01.833762 ip-10-0-129-105 kubenswrapper[2556]: E0730 20:26:01.833744 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0eb8520ad36974bb7abb53f7e96d4ad030296a895056b629dac9642412735bd6\": container with ID starting with 0eb8520ad36974bb7abb53f7e96d4ad030296a895056b629dac9642412735bd6 not found: ID does not exist" containerID="0eb8520ad36974bb7abb53f7e96d4ad030296a895056b629dac9642412735bd6" Jul 30 20:26:01.833820 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:01.833768 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eb8520ad36974bb7abb53f7e96d4ad030296a895056b629dac9642412735bd6"} err="failed to get container status \"0eb8520ad36974bb7abb53f7e96d4ad030296a895056b629dac9642412735bd6\": rpc error: code = NotFound desc = could not find container \"0eb8520ad36974bb7abb53f7e96d4ad030296a895056b629dac9642412735bd6\": container with ID starting with 0eb8520ad36974bb7abb53f7e96d4ad030296a895056b629dac9642412735bd6 not found: ID does not exist" Jul 30 20:26:01.837471 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:01.837448 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-23c617-566ff8cd48-ls9bw"] Jul 30 20:26:01.841745 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:01.841726 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-23c617-566ff8cd48-ls9bw"] Jul 30 20:26:01.852772 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:01.852755 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-23c617-5755f6dcd8-qzdn8"] Jul 30 20:26:01.856336 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:01.856319 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-23c617-5755f6dcd8-qzdn8"] Jul 30 20:26:01.886750 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:01.886726 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzxhz\" (UniqueName: \"kubernetes.io/projected/b7c92472-b77d-4d3b-8173-81f739cefcd0-kube-api-access-jzxhz\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 20:26:01.886750 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:01.886744 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rswvl\" (UniqueName: \"kubernetes.io/projected/ddfc42b4-cc19-4774-b5c9-fe2a628ef239-kube-api-access-rswvl\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 20:26:01.886854 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:01.886754 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddfc42b4-cc19-4774-b5c9-fe2a628ef239-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 20:26:01.886854 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:01.886762 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7c92472-b77d-4d3b-8173-81f739cefcd0-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 20:26:02.324596 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:02.324558 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7c92472-b77d-4d3b-8173-81f739cefcd0" path="/var/lib/kubelet/pods/b7c92472-b77d-4d3b-8173-81f739cefcd0/volumes" Jul 30 20:26:02.324931 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:02.324919 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddfc42b4-cc19-4774-b5c9-fe2a628ef239" path="/var/lib/kubelet/pods/ddfc42b4-cc19-4774-b5c9-fe2a628ef239/volumes" Jul 30 20:26:27.058608 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:27.058571 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0c599c-c649cc66d-r5zjv"] Jul 30 20:26:27.059072 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:27.058824 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-0c599c-c649cc66d-r5zjv" podUID="32caabdc-5d4c-4a3b-965b-edc180546f1a" containerName="payload-processing" containerID="cri-o://b4066c05082096ff7c2488deb36cccd3c5562afab6d9f3c5371be7dd40e70954" gracePeriod=30 Jul 30 20:26:27.064690 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:27.064664 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0c599c-66f5b555bd-f4cd2"] Jul 30 20:26:27.064869 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:27.064848 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c599c-66f5b555bd-f4cd2" podUID="94b47a93-6faa-43fd-a35d-d951180afba2" containerName="payload-pre-processing" containerID="cri-o://6647f28f936c17031168fd25c6e22ad8eaf0c65026df4b11e24b49e3fdab5686" gracePeriod=30 Jul 30 20:26:27.356868 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:27.356838 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c599c-66f5b555bd-f4cd2" Jul 30 20:26:27.359585 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:27.359567 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0c599c-c649cc66d-r5zjv" Jul 30 20:26:27.503959 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:27.503927 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8bm7x\" (UniqueName: \"kubernetes.io/projected/32caabdc-5d4c-4a3b-965b-edc180546f1a-kube-api-access-8bm7x\") pod \"32caabdc-5d4c-4a3b-965b-edc180546f1a\" (UID: \"32caabdc-5d4c-4a3b-965b-edc180546f1a\") " Jul 30 20:26:27.504118 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:27.504024 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94b47a93-6faa-43fd-a35d-d951180afba2-plugins-config-volume\") pod \"94b47a93-6faa-43fd-a35d-d951180afba2\" (UID: \"94b47a93-6faa-43fd-a35d-d951180afba2\") " Jul 30 20:26:27.504118 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:27.504052 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32caabdc-5d4c-4a3b-965b-edc180546f1a-plugins-config-volume\") pod \"32caabdc-5d4c-4a3b-965b-edc180546f1a\" (UID: \"32caabdc-5d4c-4a3b-965b-edc180546f1a\") " Jul 30 20:26:27.504245 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:27.504205 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lcn89\" (UniqueName: \"kubernetes.io/projected/94b47a93-6faa-43fd-a35d-d951180afba2-kube-api-access-lcn89\") pod \"94b47a93-6faa-43fd-a35d-d951180afba2\" (UID: \"94b47a93-6faa-43fd-a35d-d951180afba2\") " Jul 30 20:26:27.504457 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:27.504421 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94b47a93-6faa-43fd-a35d-d951180afba2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "94b47a93-6faa-43fd-a35d-d951180afba2" (UID: "94b47a93-6faa-43fd-a35d-d951180afba2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 20:26:27.504601 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:27.504507 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32caabdc-5d4c-4a3b-965b-edc180546f1a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "32caabdc-5d4c-4a3b-965b-edc180546f1a" (UID: "32caabdc-5d4c-4a3b-965b-edc180546f1a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 20:26:27.504601 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:27.504570 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94b47a93-6faa-43fd-a35d-d951180afba2-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 20:26:27.505985 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:27.505965 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32caabdc-5d4c-4a3b-965b-edc180546f1a-kube-api-access-8bm7x" (OuterVolumeSpecName: "kube-api-access-8bm7x") pod "32caabdc-5d4c-4a3b-965b-edc180546f1a" (UID: "32caabdc-5d4c-4a3b-965b-edc180546f1a"). InnerVolumeSpecName "kube-api-access-8bm7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:26:27.506146 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:27.506128 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94b47a93-6faa-43fd-a35d-d951180afba2-kube-api-access-lcn89" (OuterVolumeSpecName: "kube-api-access-lcn89") pod "94b47a93-6faa-43fd-a35d-d951180afba2" (UID: "94b47a93-6faa-43fd-a35d-d951180afba2"). InnerVolumeSpecName "kube-api-access-lcn89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:26:27.605820 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:27.605750 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcn89\" (UniqueName: \"kubernetes.io/projected/94b47a93-6faa-43fd-a35d-d951180afba2-kube-api-access-lcn89\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 20:26:27.605820 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:27.605777 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bm7x\" (UniqueName: \"kubernetes.io/projected/32caabdc-5d4c-4a3b-965b-edc180546f1a-kube-api-access-8bm7x\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 20:26:27.605820 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:27.605790 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32caabdc-5d4c-4a3b-965b-edc180546f1a-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 20:26:27.949143 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:27.949106 2556 generic.go:356] "Generic (PLEG): container finished" podID="94b47a93-6faa-43fd-a35d-d951180afba2" containerID="6647f28f936c17031168fd25c6e22ad8eaf0c65026df4b11e24b49e3fdab5686" exitCode=0 Jul 30 20:26:27.949335 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:27.949154 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c599c-66f5b555bd-f4cd2" Jul 30 20:26:27.949335 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:27.949190 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c599c-66f5b555bd-f4cd2" event={"ID":"94b47a93-6faa-43fd-a35d-d951180afba2","Type":"ContainerDied","Data":"6647f28f936c17031168fd25c6e22ad8eaf0c65026df4b11e24b49e3fdab5686"} Jul 30 20:26:27.949335 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:27.949251 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0c599c-66f5b555bd-f4cd2" event={"ID":"94b47a93-6faa-43fd-a35d-d951180afba2","Type":"ContainerDied","Data":"01ab1f9d43f73af74f8112056d2c209baeca8306e745d9effe1969ef543cd801"} Jul 30 20:26:27.949335 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:27.949286 2556 scope.go:117] "RemoveContainer" containerID="6647f28f936c17031168fd25c6e22ad8eaf0c65026df4b11e24b49e3fdab5686" Jul 30 20:26:27.950710 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:27.950684 2556 generic.go:356] "Generic (PLEG): container finished" podID="32caabdc-5d4c-4a3b-965b-edc180546f1a" containerID="b4066c05082096ff7c2488deb36cccd3c5562afab6d9f3c5371be7dd40e70954" exitCode=0 Jul 30 20:26:27.950831 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:27.950731 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0c599c-c649cc66d-r5zjv" Jul 30 20:26:27.950831 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:27.950745 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0c599c-c649cc66d-r5zjv" event={"ID":"32caabdc-5d4c-4a3b-965b-edc180546f1a","Type":"ContainerDied","Data":"b4066c05082096ff7c2488deb36cccd3c5562afab6d9f3c5371be7dd40e70954"} Jul 30 20:26:27.950831 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:27.950784 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0c599c-c649cc66d-r5zjv" event={"ID":"32caabdc-5d4c-4a3b-965b-edc180546f1a","Type":"ContainerDied","Data":"0ef4e2c79a4ac78592452c01722f62c524945bc82fdd9e789f321fc50add0d5d"} Jul 30 20:26:27.966231 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:27.966212 2556 scope.go:117] "RemoveContainer" containerID="6647f28f936c17031168fd25c6e22ad8eaf0c65026df4b11e24b49e3fdab5686" Jul 30 20:26:27.966574 ip-10-0-129-105 kubenswrapper[2556]: E0730 20:26:27.966552 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6647f28f936c17031168fd25c6e22ad8eaf0c65026df4b11e24b49e3fdab5686\": container with ID starting with 6647f28f936c17031168fd25c6e22ad8eaf0c65026df4b11e24b49e3fdab5686 not found: ID does not exist" containerID="6647f28f936c17031168fd25c6e22ad8eaf0c65026df4b11e24b49e3fdab5686" Jul 30 20:26:27.966684 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:27.966583 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6647f28f936c17031168fd25c6e22ad8eaf0c65026df4b11e24b49e3fdab5686"} err="failed to get container status \"6647f28f936c17031168fd25c6e22ad8eaf0c65026df4b11e24b49e3fdab5686\": rpc error: code = NotFound desc = could not find container \"6647f28f936c17031168fd25c6e22ad8eaf0c65026df4b11e24b49e3fdab5686\": container with ID starting with 6647f28f936c17031168fd25c6e22ad8eaf0c65026df4b11e24b49e3fdab5686 not found: ID does not exist" Jul 30 20:26:27.966684 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:27.966606 2556 scope.go:117] "RemoveContainer" containerID="b4066c05082096ff7c2488deb36cccd3c5562afab6d9f3c5371be7dd40e70954" Jul 30 20:26:27.976711 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:27.976558 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0c599c-66f5b555bd-f4cd2"] Jul 30 20:26:27.976910 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:27.976845 2556 scope.go:117] "RemoveContainer" containerID="b4066c05082096ff7c2488deb36cccd3c5562afab6d9f3c5371be7dd40e70954" Jul 30 20:26:27.977229 ip-10-0-129-105 kubenswrapper[2556]: E0730 20:26:27.977173 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4066c05082096ff7c2488deb36cccd3c5562afab6d9f3c5371be7dd40e70954\": container with ID starting with b4066c05082096ff7c2488deb36cccd3c5562afab6d9f3c5371be7dd40e70954 not found: ID does not exist" containerID="b4066c05082096ff7c2488deb36cccd3c5562afab6d9f3c5371be7dd40e70954" Jul 30 20:26:27.977229 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:27.977203 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4066c05082096ff7c2488deb36cccd3c5562afab6d9f3c5371be7dd40e70954"} err="failed to get container status \"b4066c05082096ff7c2488deb36cccd3c5562afab6d9f3c5371be7dd40e70954\": rpc error: code = NotFound desc = could not find container \"b4066c05082096ff7c2488deb36cccd3c5562afab6d9f3c5371be7dd40e70954\": container with ID starting with b4066c05082096ff7c2488deb36cccd3c5562afab6d9f3c5371be7dd40e70954 not found: ID does not exist" Jul 30 20:26:27.981440 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:27.981422 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0c599c-66f5b555bd-f4cd2"] Jul 30 20:26:27.990616 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:27.990596 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0c599c-c649cc66d-r5zjv"] Jul 30 20:26:27.997831 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:27.997812 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0c599c-c649cc66d-r5zjv"] Jul 30 20:26:28.324578 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:28.324515 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32caabdc-5d4c-4a3b-965b-edc180546f1a" path="/var/lib/kubelet/pods/32caabdc-5d4c-4a3b-965b-edc180546f1a/volumes" Jul 30 20:26:28.324901 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:26:28.324863 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94b47a93-6faa-43fd-a35d-d951180afba2" path="/var/lib/kubelet/pods/94b47a93-6faa-43fd-a35d-d951180afba2/volumes" Jul 30 20:27:40.996911 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:27:40.996822 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d67d52-58cc76fb6d-wmg7n"] Jul 30 20:27:40.997439 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:27:40.997080 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d67d52-58cc76fb6d-wmg7n" podUID="0447a80f-534f-4c9b-ae4b-b077c7882b28" containerName="payload-pre-processing" containerID="cri-o://8dd76b47c5ca6adb5ad90b9b09cd592152a6bc1491e12403b3882ba5df4ea248" gracePeriod=30 Jul 30 20:27:41.264920 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:27:41.264893 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d67d52-58cc76fb6d-wmg7n" Jul 30 20:27:41.318088 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:27:41.318053 2556 generic.go:356] "Generic (PLEG): container finished" podID="0447a80f-534f-4c9b-ae4b-b077c7882b28" containerID="8dd76b47c5ca6adb5ad90b9b09cd592152a6bc1491e12403b3882ba5df4ea248" exitCode=0 Jul 30 20:27:41.318249 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:27:41.318108 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d67d52-58cc76fb6d-wmg7n" Jul 30 20:27:41.318249 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:27:41.318131 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d67d52-58cc76fb6d-wmg7n" event={"ID":"0447a80f-534f-4c9b-ae4b-b077c7882b28","Type":"ContainerDied","Data":"8dd76b47c5ca6adb5ad90b9b09cd592152a6bc1491e12403b3882ba5df4ea248"} Jul 30 20:27:41.318249 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:27:41.318178 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d67d52-58cc76fb6d-wmg7n" event={"ID":"0447a80f-534f-4c9b-ae4b-b077c7882b28","Type":"ContainerDied","Data":"f02e5c56da13322b3bcbcd03bfad95d8a2418feacaa4ca821c540e35654b35e8"} Jul 30 20:27:41.318249 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:27:41.318206 2556 scope.go:117] "RemoveContainer" containerID="8dd76b47c5ca6adb5ad90b9b09cd592152a6bc1491e12403b3882ba5df4ea248" Jul 30 20:27:41.327853 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:27:41.327831 2556 scope.go:117] "RemoveContainer" containerID="8dd76b47c5ca6adb5ad90b9b09cd592152a6bc1491e12403b3882ba5df4ea248" Jul 30 20:27:41.328131 ip-10-0-129-105 kubenswrapper[2556]: E0730 20:27:41.328106 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dd76b47c5ca6adb5ad90b9b09cd592152a6bc1491e12403b3882ba5df4ea248\": container with ID starting with 8dd76b47c5ca6adb5ad90b9b09cd592152a6bc1491e12403b3882ba5df4ea248 not found: ID does not exist" containerID="8dd76b47c5ca6adb5ad90b9b09cd592152a6bc1491e12403b3882ba5df4ea248" Jul 30 20:27:41.328219 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:27:41.328145 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dd76b47c5ca6adb5ad90b9b09cd592152a6bc1491e12403b3882ba5df4ea248"} err="failed to get container status \"8dd76b47c5ca6adb5ad90b9b09cd592152a6bc1491e12403b3882ba5df4ea248\": rpc error: code = NotFound desc = could not find container \"8dd76b47c5ca6adb5ad90b9b09cd592152a6bc1491e12403b3882ba5df4ea248\": container with ID starting with 8dd76b47c5ca6adb5ad90b9b09cd592152a6bc1491e12403b3882ba5df4ea248 not found: ID does not exist" Jul 30 20:27:41.341556 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:27:41.341532 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64zdc\" (UniqueName: \"kubernetes.io/projected/0447a80f-534f-4c9b-ae4b-b077c7882b28-kube-api-access-64zdc\") pod \"0447a80f-534f-4c9b-ae4b-b077c7882b28\" (UID: \"0447a80f-534f-4c9b-ae4b-b077c7882b28\") " Jul 30 20:27:41.341662 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:27:41.341631 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0447a80f-534f-4c9b-ae4b-b077c7882b28-plugins-config-volume\") pod \"0447a80f-534f-4c9b-ae4b-b077c7882b28\" (UID: \"0447a80f-534f-4c9b-ae4b-b077c7882b28\") " Jul 30 20:27:41.341968 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:27:41.341946 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0447a80f-534f-4c9b-ae4b-b077c7882b28-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0447a80f-534f-4c9b-ae4b-b077c7882b28" (UID: "0447a80f-534f-4c9b-ae4b-b077c7882b28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 20:27:41.343345 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:27:41.343322 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0447a80f-534f-4c9b-ae4b-b077c7882b28-kube-api-access-64zdc" (OuterVolumeSpecName: "kube-api-access-64zdc") pod "0447a80f-534f-4c9b-ae4b-b077c7882b28" (UID: "0447a80f-534f-4c9b-ae4b-b077c7882b28"). InnerVolumeSpecName "kube-api-access-64zdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:27:41.442959 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:27:41.442933 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64zdc\" (UniqueName: \"kubernetes.io/projected/0447a80f-534f-4c9b-ae4b-b077c7882b28-kube-api-access-64zdc\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 20:27:41.442959 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:27:41.442958 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0447a80f-534f-4c9b-ae4b-b077c7882b28-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 20:27:41.639593 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:27:41.639567 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d67d52-58cc76fb6d-wmg7n"] Jul 30 20:27:41.643208 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:27:41.643186 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d67d52-58cc76fb6d-wmg7n"] Jul 30 20:27:42.189546 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:27:42.189500 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d67d52-58cc76fb6d-wmg7n" podUID="0447a80f-534f-4c9b-ae4b-b077c7882b28" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.66:9004: i/o timeout" Jul 30 20:27:42.325276 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:27:42.325246 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0447a80f-534f-4c9b-ae4b-b077c7882b28" path="/var/lib/kubelet/pods/0447a80f-534f-4c9b-ae4b-b077c7882b28/volumes" Jul 30 20:27:58.277783 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:27:58.277746 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-ecda00-6d5df66494-nkq2b"] Jul 30 20:27:58.278433 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:27:58.278403 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-ecda00-6d5df66494-nkq2b" podUID="a6e6af4a-dc00-4746-b1d8-9a3541cc880a" containerName="payload-processing" containerID="cri-o://28100c8976e96f8ddee2927cbb642ebcb8de6dbbd8f52ba6f12e0260b135a4a6" gracePeriod=30 Jul 30 20:27:58.286868 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:27:58.286832 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-ecda00-6bdfb65bb8-kdmpc"] Jul 30 20:27:58.287595 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:27:58.287040 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ecda00-6bdfb65bb8-kdmpc" podUID="d902f0c2-c297-4956-857e-63a11ffcae73" containerName="payload-pre-processing" containerID="cri-o://1d5bb43118cdf0f1732bc77f78337e6a07b5de176491b3748d19748c8302f5b5" gracePeriod=30 Jul 30 20:27:58.407982 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:27:58.407945 2556 generic.go:356] "Generic (PLEG): container finished" podID="a6e6af4a-dc00-4746-b1d8-9a3541cc880a" containerID="28100c8976e96f8ddee2927cbb642ebcb8de6dbbd8f52ba6f12e0260b135a4a6" exitCode=0 Jul 30 20:27:58.408138 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:27:58.408089 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-ecda00-6d5df66494-nkq2b" event={"ID":"a6e6af4a-dc00-4746-b1d8-9a3541cc880a","Type":"ContainerDied","Data":"28100c8976e96f8ddee2927cbb642ebcb8de6dbbd8f52ba6f12e0260b135a4a6"} Jul 30 20:27:58.410099 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:27:58.410068 2556 generic.go:356] "Generic (PLEG): container finished" podID="d902f0c2-c297-4956-857e-63a11ffcae73" containerID="1d5bb43118cdf0f1732bc77f78337e6a07b5de176491b3748d19748c8302f5b5" exitCode=0 Jul 30 20:27:58.410229 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:27:58.410156 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ecda00-6bdfb65bb8-kdmpc" event={"ID":"d902f0c2-c297-4956-857e-63a11ffcae73","Type":"ContainerDied","Data":"1d5bb43118cdf0f1732bc77f78337e6a07b5de176491b3748d19748c8302f5b5"} Jul 30 20:27:58.574356 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:27:58.574334 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-ecda00-6d5df66494-nkq2b" Jul 30 20:27:58.577157 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:27:58.577138 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ecda00-6bdfb65bb8-kdmpc" Jul 30 20:27:58.687929 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:27:58.687898 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lwq8g\" (UniqueName: \"kubernetes.io/projected/d902f0c2-c297-4956-857e-63a11ffcae73-kube-api-access-lwq8g\") pod \"d902f0c2-c297-4956-857e-63a11ffcae73\" (UID: \"d902f0c2-c297-4956-857e-63a11ffcae73\") " Jul 30 20:27:58.688075 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:27:58.687946 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbnrs\" (UniqueName: \"kubernetes.io/projected/a6e6af4a-dc00-4746-b1d8-9a3541cc880a-kube-api-access-lbnrs\") pod \"a6e6af4a-dc00-4746-b1d8-9a3541cc880a\" (UID: \"a6e6af4a-dc00-4746-b1d8-9a3541cc880a\") " Jul 30 20:27:58.688075 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:27:58.687977 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d902f0c2-c297-4956-857e-63a11ffcae73-plugins-config-volume\") pod \"d902f0c2-c297-4956-857e-63a11ffcae73\" (UID: \"d902f0c2-c297-4956-857e-63a11ffcae73\") " Jul 30 20:27:58.688075 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:27:58.688055 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6e6af4a-dc00-4746-b1d8-9a3541cc880a-plugins-config-volume\") pod \"a6e6af4a-dc00-4746-b1d8-9a3541cc880a\" (UID: \"a6e6af4a-dc00-4746-b1d8-9a3541cc880a\") " Jul 30 20:27:58.688460 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:27:58.688435 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d902f0c2-c297-4956-857e-63a11ffcae73-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d902f0c2-c297-4956-857e-63a11ffcae73" (UID: "d902f0c2-c297-4956-857e-63a11ffcae73"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 20:27:58.688583 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:27:58.688480 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6e6af4a-dc00-4746-b1d8-9a3541cc880a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a6e6af4a-dc00-4746-b1d8-9a3541cc880a" (UID: "a6e6af4a-dc00-4746-b1d8-9a3541cc880a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 20:27:58.689947 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:27:58.689923 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6e6af4a-dc00-4746-b1d8-9a3541cc880a-kube-api-access-lbnrs" (OuterVolumeSpecName: "kube-api-access-lbnrs") pod "a6e6af4a-dc00-4746-b1d8-9a3541cc880a" (UID: "a6e6af4a-dc00-4746-b1d8-9a3541cc880a"). InnerVolumeSpecName "kube-api-access-lbnrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:27:58.690000 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:27:58.689947 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d902f0c2-c297-4956-857e-63a11ffcae73-kube-api-access-lwq8g" (OuterVolumeSpecName: "kube-api-access-lwq8g") pod "d902f0c2-c297-4956-857e-63a11ffcae73" (UID: "d902f0c2-c297-4956-857e-63a11ffcae73"). InnerVolumeSpecName "kube-api-access-lwq8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:27:58.789197 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:27:58.789174 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwq8g\" (UniqueName: \"kubernetes.io/projected/d902f0c2-c297-4956-857e-63a11ffcae73-kube-api-access-lwq8g\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 20:27:58.789197 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:27:58.789194 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbnrs\" (UniqueName: \"kubernetes.io/projected/a6e6af4a-dc00-4746-b1d8-9a3541cc880a-kube-api-access-lbnrs\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 20:27:58.789333 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:27:58.789203 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d902f0c2-c297-4956-857e-63a11ffcae73-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 20:27:58.789333 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:27:58.789213 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6e6af4a-dc00-4746-b1d8-9a3541cc880a-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 20:27:59.416024 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:27:59.415999 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-ecda00-6d5df66494-nkq2b" Jul 30 20:27:59.416444 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:27:59.416034 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-ecda00-6d5df66494-nkq2b" event={"ID":"a6e6af4a-dc00-4746-b1d8-9a3541cc880a","Type":"ContainerDied","Data":"75283bf495cb226b4347da25649b21ced3cc88de7bda94cbb22f753f6776c90a"} Jul 30 20:27:59.416444 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:27:59.416102 2556 scope.go:117] "RemoveContainer" containerID="28100c8976e96f8ddee2927cbb642ebcb8de6dbbd8f52ba6f12e0260b135a4a6" Jul 30 20:27:59.417434 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:27:59.417412 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ecda00-6bdfb65bb8-kdmpc" event={"ID":"d902f0c2-c297-4956-857e-63a11ffcae73","Type":"ContainerDied","Data":"ef129ebfa67201360cf001a1a977572f29f0d2b3967bfe04a5daadfc4bcdd6cc"} Jul 30 20:27:59.417534 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:27:59.417422 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ecda00-6bdfb65bb8-kdmpc" Jul 30 20:27:59.427473 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:27:59.426907 2556 scope.go:117] "RemoveContainer" containerID="1d5bb43118cdf0f1732bc77f78337e6a07b5de176491b3748d19748c8302f5b5" Jul 30 20:27:59.442676 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:27:59.442657 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-ecda00-6bdfb65bb8-kdmpc"] Jul 30 20:27:59.446784 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:27:59.446763 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-ecda00-6bdfb65bb8-kdmpc"] Jul 30 20:27:59.461427 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:27:59.459084 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-ecda00-6d5df66494-nkq2b"] Jul 30 20:27:59.464305 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:27:59.464279 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-ecda00-6d5df66494-nkq2b"] Jul 30 20:28:00.324499 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:00.324468 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6e6af4a-dc00-4746-b1d8-9a3541cc880a" path="/var/lib/kubelet/pods/a6e6af4a-dc00-4746-b1d8-9a3541cc880a/volumes" Jul 30 20:28:00.324829 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:00.324817 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d902f0c2-c297-4956-857e-63a11ffcae73" path="/var/lib/kubelet/pods/d902f0c2-c297-4956-857e-63a11ffcae73/volumes" Jul 30 20:28:15.794617 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:15.794587 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-a769e5-6fd4dbbf9d-wzpq9"] Jul 30 20:28:15.795055 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:15.794784 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-a769e5-6fd4dbbf9d-wzpq9" podUID="645b7fef-fa10-4957-91af-a5c1fef25806" containerName="payload-processing" containerID="cri-o://6e132fc8d263f68c90b558f7e09d15acdc389f426ba515726314abf966f96f64" gracePeriod=30 Jul 30 20:28:15.800967 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:15.800939 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-a769e5-5c697f9788-vmjhv"] Jul 30 20:28:15.801177 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:15.801141 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-a769e5-5c697f9788-vmjhv" podUID="7f759bb7-6a34-4043-bf5f-78d69c7b305f" containerName="payload-pre-processing" containerID="cri-o://cf0c95638853747da933a784bff4c613a9663bdba3079894179d7120d9ebfc52" gracePeriod=30 Jul 30 20:28:16.061863 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:16.061839 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-a769e5-6fd4dbbf9d-wzpq9" Jul 30 20:28:16.093687 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:16.093633 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-a769e5-5c697f9788-vmjhv" Jul 30 20:28:16.139708 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:16.139682 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6jmg\" (UniqueName: \"kubernetes.io/projected/645b7fef-fa10-4957-91af-a5c1fef25806-kube-api-access-s6jmg\") pod \"645b7fef-fa10-4957-91af-a5c1fef25806\" (UID: \"645b7fef-fa10-4957-91af-a5c1fef25806\") " Jul 30 20:28:16.139863 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:16.139763 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/645b7fef-fa10-4957-91af-a5c1fef25806-plugins-config-volume\") pod \"645b7fef-fa10-4957-91af-a5c1fef25806\" (UID: \"645b7fef-fa10-4957-91af-a5c1fef25806\") " Jul 30 20:28:16.140137 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:16.140116 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/645b7fef-fa10-4957-91af-a5c1fef25806-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "645b7fef-fa10-4957-91af-a5c1fef25806" (UID: "645b7fef-fa10-4957-91af-a5c1fef25806"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 20:28:16.141621 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:16.141603 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/645b7fef-fa10-4957-91af-a5c1fef25806-kube-api-access-s6jmg" (OuterVolumeSpecName: "kube-api-access-s6jmg") pod "645b7fef-fa10-4957-91af-a5c1fef25806" (UID: "645b7fef-fa10-4957-91af-a5c1fef25806"). InnerVolumeSpecName "kube-api-access-s6jmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:28:16.241100 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:16.241074 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f759bb7-6a34-4043-bf5f-78d69c7b305f-plugins-config-volume\") pod \"7f759bb7-6a34-4043-bf5f-78d69c7b305f\" (UID: \"7f759bb7-6a34-4043-bf5f-78d69c7b305f\") " Jul 30 20:28:16.241234 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:16.241135 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-htgf9\" (UniqueName: \"kubernetes.io/projected/7f759bb7-6a34-4043-bf5f-78d69c7b305f-kube-api-access-htgf9\") pod \"7f759bb7-6a34-4043-bf5f-78d69c7b305f\" (UID: \"7f759bb7-6a34-4043-bf5f-78d69c7b305f\") " Jul 30 20:28:16.241297 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:16.241279 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6jmg\" (UniqueName: \"kubernetes.io/projected/645b7fef-fa10-4957-91af-a5c1fef25806-kube-api-access-s6jmg\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 20:28:16.241297 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:16.241292 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/645b7fef-fa10-4957-91af-a5c1fef25806-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 20:28:16.241439 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:16.241418 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f759bb7-6a34-4043-bf5f-78d69c7b305f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7f759bb7-6a34-4043-bf5f-78d69c7b305f" (UID: "7f759bb7-6a34-4043-bf5f-78d69c7b305f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 20:28:16.242984 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:16.242956 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f759bb7-6a34-4043-bf5f-78d69c7b305f-kube-api-access-htgf9" (OuterVolumeSpecName: "kube-api-access-htgf9") pod "7f759bb7-6a34-4043-bf5f-78d69c7b305f" (UID: "7f759bb7-6a34-4043-bf5f-78d69c7b305f"). InnerVolumeSpecName "kube-api-access-htgf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:28:16.342605 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:16.342586 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f759bb7-6a34-4043-bf5f-78d69c7b305f-plugins-config-volume\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 20:28:16.342678 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:16.342607 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htgf9\" (UniqueName: \"kubernetes.io/projected/7f759bb7-6a34-4043-bf5f-78d69c7b305f-kube-api-access-htgf9\") on node \"ip-10-0-129-105.ec2.internal\" DevicePath \"\"" Jul 30 20:28:16.502237 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:16.502206 2556 generic.go:356] "Generic (PLEG): container finished" podID="7f759bb7-6a34-4043-bf5f-78d69c7b305f" containerID="cf0c95638853747da933a784bff4c613a9663bdba3079894179d7120d9ebfc52" exitCode=0 Jul 30 20:28:16.502378 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:16.502252 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-a769e5-5c697f9788-vmjhv" Jul 30 20:28:16.502378 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:16.502291 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-a769e5-5c697f9788-vmjhv" event={"ID":"7f759bb7-6a34-4043-bf5f-78d69c7b305f","Type":"ContainerDied","Data":"cf0c95638853747da933a784bff4c613a9663bdba3079894179d7120d9ebfc52"} Jul 30 20:28:16.502378 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:16.502338 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-a769e5-5c697f9788-vmjhv" event={"ID":"7f759bb7-6a34-4043-bf5f-78d69c7b305f","Type":"ContainerDied","Data":"b56ff0e8e83fb7681173de3f3804e0237ffe3047c3ade8d90145577d09bb6420"} Jul 30 20:28:16.502378 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:16.502359 2556 scope.go:117] "RemoveContainer" containerID="cf0c95638853747da933a784bff4c613a9663bdba3079894179d7120d9ebfc52" Jul 30 20:28:16.503678 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:16.503655 2556 generic.go:356] "Generic (PLEG): container finished" podID="645b7fef-fa10-4957-91af-a5c1fef25806" containerID="6e132fc8d263f68c90b558f7e09d15acdc389f426ba515726314abf966f96f64" exitCode=0 Jul 30 20:28:16.503776 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:16.503701 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-a769e5-6fd4dbbf9d-wzpq9" event={"ID":"645b7fef-fa10-4957-91af-a5c1fef25806","Type":"ContainerDied","Data":"6e132fc8d263f68c90b558f7e09d15acdc389f426ba515726314abf966f96f64"} Jul 30 20:28:16.503776 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:16.503734 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-a769e5-6fd4dbbf9d-wzpq9" event={"ID":"645b7fef-fa10-4957-91af-a5c1fef25806","Type":"ContainerDied","Data":"ef343dd23a185ce985065ff7a318b9fadf8b2a0933f0173a1e0cd77f93137b33"} Jul 30 20:28:16.503776 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:16.503703 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-a769e5-6fd4dbbf9d-wzpq9" Jul 30 20:28:16.511343 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:16.511325 2556 scope.go:117] "RemoveContainer" containerID="cf0c95638853747da933a784bff4c613a9663bdba3079894179d7120d9ebfc52" Jul 30 20:28:16.511641 ip-10-0-129-105 kubenswrapper[2556]: E0730 20:28:16.511623 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf0c95638853747da933a784bff4c613a9663bdba3079894179d7120d9ebfc52\": container with ID starting with cf0c95638853747da933a784bff4c613a9663bdba3079894179d7120d9ebfc52 not found: ID does not exist" containerID="cf0c95638853747da933a784bff4c613a9663bdba3079894179d7120d9ebfc52" Jul 30 20:28:16.511719 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:16.511650 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf0c95638853747da933a784bff4c613a9663bdba3079894179d7120d9ebfc52"} err="failed to get container status \"cf0c95638853747da933a784bff4c613a9663bdba3079894179d7120d9ebfc52\": rpc error: code = NotFound desc = could not find container \"cf0c95638853747da933a784bff4c613a9663bdba3079894179d7120d9ebfc52\": container with ID starting with cf0c95638853747da933a784bff4c613a9663bdba3079894179d7120d9ebfc52 not found: ID does not exist" Jul 30 20:28:16.511719 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:16.511667 2556 scope.go:117] "RemoveContainer" containerID="6e132fc8d263f68c90b558f7e09d15acdc389f426ba515726314abf966f96f64" Jul 30 20:28:16.519168 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:16.519150 2556 scope.go:117] "RemoveContainer" containerID="6e132fc8d263f68c90b558f7e09d15acdc389f426ba515726314abf966f96f64" Jul 30 20:28:16.519428 ip-10-0-129-105 kubenswrapper[2556]: E0730 20:28:16.519409 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e132fc8d263f68c90b558f7e09d15acdc389f426ba515726314abf966f96f64\": container with ID starting with 6e132fc8d263f68c90b558f7e09d15acdc389f426ba515726314abf966f96f64 not found: ID does not exist" containerID="6e132fc8d263f68c90b558f7e09d15acdc389f426ba515726314abf966f96f64" Jul 30 20:28:16.519495 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:16.519433 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e132fc8d263f68c90b558f7e09d15acdc389f426ba515726314abf966f96f64"} err="failed to get container status \"6e132fc8d263f68c90b558f7e09d15acdc389f426ba515726314abf966f96f64\": rpc error: code = NotFound desc = could not find container \"6e132fc8d263f68c90b558f7e09d15acdc389f426ba515726314abf966f96f64\": container with ID starting with 6e132fc8d263f68c90b558f7e09d15acdc389f426ba515726314abf966f96f64 not found: ID does not exist" Jul 30 20:28:16.522207 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:16.522187 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-a769e5-6fd4dbbf9d-wzpq9"] Jul 30 20:28:16.526176 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:16.526157 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-a769e5-6fd4dbbf9d-wzpq9"] Jul 30 20:28:16.536011 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:16.535989 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-a769e5-5c697f9788-vmjhv"] Jul 30 20:28:16.538441 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:16.538423 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-a769e5-5c697f9788-vmjhv"] Jul 30 20:28:18.323813 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:18.323780 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="645b7fef-fa10-4957-91af-a5c1fef25806" path="/var/lib/kubelet/pods/645b7fef-fa10-4957-91af-a5c1fef25806/volumes" Jul 30 20:28:18.324181 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:18.324122 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f759bb7-6a34-4043-bf5f-78d69c7b305f" path="/var/lib/kubelet/pods/7f759bb7-6a34-4043-bf5f-78d69c7b305f/volumes" Jul 30 20:28:28.633750 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:28.633716 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-ccptj_86ec7077-e86c-4f03-ac13-8bded5e52536/manager/0.log" Jul 30 20:28:28.752969 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:28.752939 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-vrmx5_429bcbb5-b835-4e1a-9c76-f4512160f42e/manager/0.log" Jul 30 20:28:28.991351 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:28.991285 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-ldgsc_8684d264-ac83-4de0-90a7-da2723d55c46/server/0.log" Jul 30 20:28:29.112398 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:29.112360 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-ntx4s_ab7a30d6-2b2d-48f9-87d2-b773a81e2dfe/manager/0.log" Jul 30 20:28:29.474763 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:29.474733 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76fdfbdbf4-w5r56_0604020a-98cf-4d6c-8ee7-a12cbda5f73d/copy-manifests/0.log" Jul 30 20:28:29.486891 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:29.486869 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76fdfbdbf4-w5r56_0604020a-98cf-4d6c-8ee7-a12cbda5f73d/manager/0.log" Jul 30 20:28:31.699228 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:31.699198 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-vr5js_5c5dee6c-0969-4dd4-8d79-0417dba7c5ad/manager/0.log" Jul 30 20:28:32.393983 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:32.393955 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-lbpw5_90883584-bc52-4465-a02f-cf5f565636d0/manager/0.log" Jul 30 20:28:32.866737 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:32.866662 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-6ftv2_fce6373e-9c39-45c0-9d46-a9c42a950783/discovery/0.log" Jul 30 20:28:32.978292 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:32.978264 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b4b6cc975-89d9s_b68a3551-b660-476a-84e5-66cb8328fa6b/kube-auth-proxy/0.log" Jul 30 20:28:33.325018 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:33.324987 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-w2gbm_8a9ee543-86c7-4d8e-ad53-cef7ece4bbf1/payload-pre-processing/0.log" Jul 30 20:28:33.459749 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:33.459718 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-knk59_bea701bd-a892-44cf-bedf-cbb6bd64bb37/payload-processing/0.log" Jul 30 20:28:34.039310 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:34.039284 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-g9xdg_855f848a-64ad-4bae-9216-132d39196bbf/main/0.log" Jul 30 20:28:34.259398 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:34.259354 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdqrz9_6e28b750-b501-4578-b0b6-c9aef9c76eec/main/0.log" Jul 30 20:28:34.373334 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:34.373305 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-t7vh2_b89d74c6-dbd3-4435-9834-689783959c45/main/0.log" Jul 30 20:28:42.639044 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:42.639016 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6tpv8_8f32c8f1-7bf6-4b84-8205-555365c55861/global-pull-secret-syncer/0.log" Jul 30 20:28:42.834871 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:42.834839 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ttglk_fb61e5ec-e983-407c-962c-1552688cbe5c/konnectivity-agent/0.log" Jul 30 20:28:42.855302 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:42.855275 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-105.ec2.internal_1728156c8efd1097c5a710ba4d71fa3e/haproxy/0.log" Jul 30 20:28:48.087484 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:48.087408 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-vr5js_5c5dee6c-0969-4dd4-8d79-0417dba7c5ad/manager/0.log" Jul 30 20:28:48.365108 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:48.365019 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-lbpw5_90883584-bc52-4465-a02f-cf5f565636d0/manager/0.log" Jul 30 20:28:50.037263 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:50.037194 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-58k7q_35a74185-af33-4abd-9973-94dc2c4e8bb9/kube-state-metrics/0.log" Jul 30 20:28:50.072493 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:50.072473 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-58k7q_35a74185-af33-4abd-9973-94dc2c4e8bb9/kube-rbac-proxy-main/0.log" Jul 30 20:28:50.132493 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:50.132475 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-58k7q_35a74185-af33-4abd-9973-94dc2c4e8bb9/kube-rbac-proxy-self/0.log" Jul 30 20:28:50.279980 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:50.279959 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-6dffbbd67-hpn6r_ed0bae37-0a50-4f34-b0b8-bf912e2f75ed/monitoring-plugin/0.log" Jul 30 20:28:50.509200 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:50.509171 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ctcb9_695d4498-ab00-4727-8f9b-7b18314070bc/node-exporter/0.log" Jul 30 20:28:50.537785 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:50.537768 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ctcb9_695d4498-ab00-4727-8f9b-7b18314070bc/kube-rbac-proxy/0.log" Jul 30 20:28:50.567993 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:50.567970 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ctcb9_695d4498-ab00-4727-8f9b-7b18314070bc/init-textfile/0.log" Jul 30 20:28:50.700776 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:50.700745 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-bk87q_c8414552-763b-40c0-9a01-f4ad8e5dd521/kube-rbac-proxy-main/0.log" Jul 30 20:28:50.732562 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:50.732545 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-bk87q_c8414552-763b-40c0-9a01-f4ad8e5dd521/kube-rbac-proxy-self/0.log" Jul 30 20:28:50.760528 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:50.760479 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-bk87q_c8414552-763b-40c0-9a01-f4ad8e5dd521/openshift-state-metrics/0.log" Jul 30 20:28:50.808168 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:50.808151 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fece5390-a1e9-4126-86bd-91af2af5ccf7/prometheus/0.log" Jul 30 20:28:50.840938 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:50.840921 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fece5390-a1e9-4126-86bd-91af2af5ccf7/config-reloader/0.log" Jul 30 20:28:50.874822 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:50.874806 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fece5390-a1e9-4126-86bd-91af2af5ccf7/thanos-sidecar/0.log" Jul 30 20:28:50.909027 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:50.909008 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fece5390-a1e9-4126-86bd-91af2af5ccf7/kube-rbac-proxy-web/0.log" Jul 30 20:28:50.946025 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:50.945997 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fece5390-a1e9-4126-86bd-91af2af5ccf7/kube-rbac-proxy/0.log" Jul 30 20:28:50.979023 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:50.979006 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fece5390-a1e9-4126-86bd-91af2af5ccf7/kube-rbac-proxy-thanos/0.log" Jul 30 20:28:51.005768 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:51.005752 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fece5390-a1e9-4126-86bd-91af2af5ccf7/init-config-reloader/0.log" Jul 30 20:28:51.036658 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:51.036610 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-758d7cb874-5wjvs_5fe53e84-a2ba-4338-8d98-162bc6a708d9/prometheus-operator/0.log" Jul 30 20:28:51.061019 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:51.061004 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-758d7cb874-5wjvs_5fe53e84-a2ba-4338-8d98-162bc6a708d9/kube-rbac-proxy/0.log" Jul 30 20:28:51.088255 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:51.088239 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-66685ff6d4-jz9fk_d1d2af65-5934-4636-8c1c-734d05a36186/prometheus-operator-admission-webhook/0.log" Jul 30 20:28:51.116137 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:51.116117 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6cbd87958-csgb5_c71030a2-fcdd-466c-8299-01c7f963c64d/telemeter-client/0.log" Jul 30 20:28:51.138744 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:51.138726 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6cbd87958-csgb5_c71030a2-fcdd-466c-8299-01c7f963c64d/reload/0.log" Jul 30 20:28:51.164477 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:51.164458 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6cbd87958-csgb5_c71030a2-fcdd-466c-8299-01c7f963c64d/kube-rbac-proxy/0.log" Jul 30 20:28:51.196672 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:51.196656 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-fd44f7995-8wknj_b0261e45-1760-4a3e-925d-fe0d63dc7635/thanos-query/0.log" Jul 30 20:28:51.225648 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:51.225630 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-fd44f7995-8wknj_b0261e45-1760-4a3e-925d-fe0d63dc7635/kube-rbac-proxy-web/0.log" Jul 30 20:28:51.248374 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:51.248342 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-fd44f7995-8wknj_b0261e45-1760-4a3e-925d-fe0d63dc7635/kube-rbac-proxy/0.log" Jul 30 20:28:51.274759 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:51.274743 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-fd44f7995-8wknj_b0261e45-1760-4a3e-925d-fe0d63dc7635/prom-label-proxy/0.log" Jul 30 20:28:51.303338 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:51.303287 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-fd44f7995-8wknj_b0261e45-1760-4a3e-925d-fe0d63dc7635/kube-rbac-proxy-rules/0.log" Jul 30 20:28:51.334669 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:51.334649 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-fd44f7995-8wknj_b0261e45-1760-4a3e-925d-fe0d63dc7635/kube-rbac-proxy-metrics/0.log" Jul 30 20:28:52.822154 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:52.822123 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fmzs6/perf-node-gather-daemonset-gr2dp"] Jul 30 20:28:52.822546 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:52.822534 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f759bb7-6a34-4043-bf5f-78d69c7b305f" containerName="payload-pre-processing" Jul 30 20:28:52.822546 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:52.822545 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f759bb7-6a34-4043-bf5f-78d69c7b305f" containerName="payload-pre-processing" Jul 30 20:28:52.822618 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:52.822553 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7c92472-b77d-4d3b-8173-81f739cefcd0" containerName="payload-pre-processing" Jul 30 20:28:52.822618 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:52.822558 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7c92472-b77d-4d3b-8173-81f739cefcd0" containerName="payload-pre-processing" Jul 30 20:28:52.822618 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:52.822567 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6e6af4a-dc00-4746-b1d8-9a3541cc880a" containerName="payload-processing" Jul 30 20:28:52.822618 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:52.822573 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6e6af4a-dc00-4746-b1d8-9a3541cc880a" containerName="payload-processing" Jul 30 20:28:52.822618 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:52.822580 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddfc42b4-cc19-4774-b5c9-fe2a628ef239" containerName="payload-processing" Jul 30 20:28:52.822618 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:52.822586 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddfc42b4-cc19-4774-b5c9-fe2a628ef239" containerName="payload-processing" Jul 30 20:28:52.822618 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:52.822594 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d902f0c2-c297-4956-857e-63a11ffcae73" containerName="payload-pre-processing" Jul 30 20:28:52.822618 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:52.822599 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d902f0c2-c297-4956-857e-63a11ffcae73" containerName="payload-pre-processing" Jul 30 20:28:52.822618 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:52.822617 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0447a80f-534f-4c9b-ae4b-b077c7882b28" containerName="payload-pre-processing" Jul 30 20:28:52.822618 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:52.822622 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="0447a80f-534f-4c9b-ae4b-b077c7882b28" containerName="payload-pre-processing" Jul 30 20:28:52.822912 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:52.822633 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="645b7fef-fa10-4957-91af-a5c1fef25806" containerName="payload-processing" Jul 30 20:28:52.822912 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:52.822638 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="645b7fef-fa10-4957-91af-a5c1fef25806" containerName="payload-processing" Jul 30 20:28:52.822912 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:52.822645 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94b47a93-6faa-43fd-a35d-d951180afba2" containerName="payload-pre-processing" Jul 30 20:28:52.822912 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:52.822650 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="94b47a93-6faa-43fd-a35d-d951180afba2" containerName="payload-pre-processing" Jul 30 20:28:52.822912 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:52.822656 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32caabdc-5d4c-4a3b-965b-edc180546f1a" containerName="payload-processing" Jul 30 20:28:52.822912 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:52.822661 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="32caabdc-5d4c-4a3b-965b-edc180546f1a" containerName="payload-processing" Jul 30 20:28:52.822912 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:52.822717 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d902f0c2-c297-4956-857e-63a11ffcae73" containerName="payload-pre-processing" Jul 30 20:28:52.822912 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:52.822724 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6e6af4a-dc00-4746-b1d8-9a3541cc880a" containerName="payload-processing" Jul 30 20:28:52.822912 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:52.822730 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="32caabdc-5d4c-4a3b-965b-edc180546f1a" containerName="payload-processing" Jul 30 20:28:52.822912 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:52.822735 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="0447a80f-534f-4c9b-ae4b-b077c7882b28" containerName="payload-pre-processing" Jul 30 20:28:52.822912 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:52.822741 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="94b47a93-6faa-43fd-a35d-d951180afba2" containerName="payload-pre-processing" Jul 30 20:28:52.822912 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:52.822746 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddfc42b4-cc19-4774-b5c9-fe2a628ef239" containerName="payload-processing" Jul 30 20:28:52.822912 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:52.822751 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f759bb7-6a34-4043-bf5f-78d69c7b305f" containerName="payload-pre-processing" Jul 30 20:28:52.822912 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:52.822757 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="645b7fef-fa10-4957-91af-a5c1fef25806" containerName="payload-processing" Jul 30 20:28:52.822912 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:52.822762 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7c92472-b77d-4d3b-8173-81f739cefcd0" containerName="payload-pre-processing" Jul 30 20:28:52.827127 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:52.827107 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fmzs6/perf-node-gather-daemonset-gr2dp" Jul 30 20:28:52.829878 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:52.829847 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fmzs6\"/\"kube-root-ca.crt\"" Jul 30 20:28:52.830232 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:52.830206 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fmzs6\"/\"default-dockercfg-xc2gz\"" Jul 30 20:28:52.830339 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:52.830219 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fmzs6\"/\"openshift-service-ca.crt\"" Jul 30 20:28:52.835103 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:52.835081 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fmzs6/perf-node-gather-daemonset-gr2dp"] Jul 30 20:28:52.848087 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:52.848066 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50feb31b-a3e0-4c54-bfdf-630cf6f9568c-sys\") pod \"perf-node-gather-daemonset-gr2dp\" (UID: \"50feb31b-a3e0-4c54-bfdf-630cf6f9568c\") " pod="openshift-must-gather-fmzs6/perf-node-gather-daemonset-gr2dp" Jul 30 20:28:52.848207 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:52.848110 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/50feb31b-a3e0-4c54-bfdf-630cf6f9568c-podres\") pod \"perf-node-gather-daemonset-gr2dp\" (UID: \"50feb31b-a3e0-4c54-bfdf-630cf6f9568c\") " pod="openshift-must-gather-fmzs6/perf-node-gather-daemonset-gr2dp" Jul 30 20:28:52.848207 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:52.848134 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/50feb31b-a3e0-4c54-bfdf-630cf6f9568c-proc\") pod \"perf-node-gather-daemonset-gr2dp\" (UID: \"50feb31b-a3e0-4c54-bfdf-630cf6f9568c\") " pod="openshift-must-gather-fmzs6/perf-node-gather-daemonset-gr2dp" Jul 30 20:28:52.848312 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:52.848219 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50feb31b-a3e0-4c54-bfdf-630cf6f9568c-lib-modules\") pod \"perf-node-gather-daemonset-gr2dp\" (UID: \"50feb31b-a3e0-4c54-bfdf-630cf6f9568c\") " pod="openshift-must-gather-fmzs6/perf-node-gather-daemonset-gr2dp" Jul 30 20:28:52.848312 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:52.848274 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8k4l\" (UniqueName: \"kubernetes.io/projected/50feb31b-a3e0-4c54-bfdf-630cf6f9568c-kube-api-access-g8k4l\") pod \"perf-node-gather-daemonset-gr2dp\" (UID: \"50feb31b-a3e0-4c54-bfdf-630cf6f9568c\") " pod="openshift-must-gather-fmzs6/perf-node-gather-daemonset-gr2dp" Jul 30 20:28:52.949180 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:52.949151 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50feb31b-a3e0-4c54-bfdf-630cf6f9568c-sys\") pod \"perf-node-gather-daemonset-gr2dp\" (UID: \"50feb31b-a3e0-4c54-bfdf-630cf6f9568c\") " pod="openshift-must-gather-fmzs6/perf-node-gather-daemonset-gr2dp" Jul 30 20:28:52.949338 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:52.949207 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/50feb31b-a3e0-4c54-bfdf-630cf6f9568c-podres\") pod \"perf-node-gather-daemonset-gr2dp\" (UID: \"50feb31b-a3e0-4c54-bfdf-630cf6f9568c\") " pod="openshift-must-gather-fmzs6/perf-node-gather-daemonset-gr2dp" Jul 30 20:28:52.949338 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:52.949231 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/50feb31b-a3e0-4c54-bfdf-630cf6f9568c-proc\") pod \"perf-node-gather-daemonset-gr2dp\" (UID: \"50feb31b-a3e0-4c54-bfdf-630cf6f9568c\") " pod="openshift-must-gather-fmzs6/perf-node-gather-daemonset-gr2dp" Jul 30 20:28:52.949338 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:52.949229 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50feb31b-a3e0-4c54-bfdf-630cf6f9568c-sys\") pod \"perf-node-gather-daemonset-gr2dp\" (UID: \"50feb31b-a3e0-4c54-bfdf-630cf6f9568c\") " pod="openshift-must-gather-fmzs6/perf-node-gather-daemonset-gr2dp" Jul 30 20:28:52.949338 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:52.949258 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50feb31b-a3e0-4c54-bfdf-630cf6f9568c-lib-modules\") pod \"perf-node-gather-daemonset-gr2dp\" (UID: \"50feb31b-a3e0-4c54-bfdf-630cf6f9568c\") " pod="openshift-must-gather-fmzs6/perf-node-gather-daemonset-gr2dp" Jul 30 20:28:52.949338 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:52.949288 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8k4l\" (UniqueName: \"kubernetes.io/projected/50feb31b-a3e0-4c54-bfdf-630cf6f9568c-kube-api-access-g8k4l\") pod \"perf-node-gather-daemonset-gr2dp\" (UID: \"50feb31b-a3e0-4c54-bfdf-630cf6f9568c\") " pod="openshift-must-gather-fmzs6/perf-node-gather-daemonset-gr2dp" Jul 30 20:28:52.949338 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:52.949325 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/50feb31b-a3e0-4c54-bfdf-630cf6f9568c-podres\") pod \"perf-node-gather-daemonset-gr2dp\" (UID: \"50feb31b-a3e0-4c54-bfdf-630cf6f9568c\") " pod="openshift-must-gather-fmzs6/perf-node-gather-daemonset-gr2dp" Jul 30 20:28:52.949338 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:52.949331 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/50feb31b-a3e0-4c54-bfdf-630cf6f9568c-proc\") pod \"perf-node-gather-daemonset-gr2dp\" (UID: \"50feb31b-a3e0-4c54-bfdf-630cf6f9568c\") " pod="openshift-must-gather-fmzs6/perf-node-gather-daemonset-gr2dp" Jul 30 20:28:52.949645 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:52.949446 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50feb31b-a3e0-4c54-bfdf-630cf6f9568c-lib-modules\") pod \"perf-node-gather-daemonset-gr2dp\" (UID: \"50feb31b-a3e0-4c54-bfdf-630cf6f9568c\") " pod="openshift-must-gather-fmzs6/perf-node-gather-daemonset-gr2dp" Jul 30 20:28:52.957206 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:52.957184 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8k4l\" (UniqueName: \"kubernetes.io/projected/50feb31b-a3e0-4c54-bfdf-630cf6f9568c-kube-api-access-g8k4l\") pod \"perf-node-gather-daemonset-gr2dp\" (UID: \"50feb31b-a3e0-4c54-bfdf-630cf6f9568c\") " pod="openshift-must-gather-fmzs6/perf-node-gather-daemonset-gr2dp" Jul 30 20:28:53.138446 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:53.138423 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fmzs6/perf-node-gather-daemonset-gr2dp" Jul 30 20:28:53.266613 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:53.266587 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fmzs6/perf-node-gather-daemonset-gr2dp"] Jul 30 20:28:53.268086 ip-10-0-129-105 kubenswrapper[2556]: W0730 20:28:53.268062 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod50feb31b_a3e0_4c54_bfdf_630cf6f9568c.slice/crio-aaefd73d5161f01bcddd77747e2c6e12fad68b1ee84dcfba06b6aaf2213a55ca WatchSource:0}: Error finding container aaefd73d5161f01bcddd77747e2c6e12fad68b1ee84dcfba06b6aaf2213a55ca: Status 404 returned error can't find the container with id aaefd73d5161f01bcddd77747e2c6e12fad68b1ee84dcfba06b6aaf2213a55ca Jul 30 20:28:53.650139 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:53.650113 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-77c974ffcf-zvqfw_68ac40f9-2822-4b96-8fb5-7d37d192bb97/download-server/0.log" Jul 30 20:28:53.688708 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:53.688676 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fmzs6/perf-node-gather-daemonset-gr2dp" event={"ID":"50feb31b-a3e0-4c54-bfdf-630cf6f9568c","Type":"ContainerStarted","Data":"50c1e8e1ba67bfa3e3649fc4208522a25ab6d931a71cfa6d3796e1256e4eece9"} Jul 30 20:28:53.688842 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:53.688731 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fmzs6/perf-node-gather-daemonset-gr2dp" Jul 30 20:28:53.688842 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:53.688747 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fmzs6/perf-node-gather-daemonset-gr2dp" event={"ID":"50feb31b-a3e0-4c54-bfdf-630cf6f9568c","Type":"ContainerStarted","Data":"aaefd73d5161f01bcddd77747e2c6e12fad68b1ee84dcfba06b6aaf2213a55ca"} Jul 30 20:28:53.703827 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:53.703789 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fmzs6/perf-node-gather-daemonset-gr2dp" podStartSLOduration=1.7037771290000001 podStartE2EDuration="1.703777129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 20:28:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:28:53.703134408 +0000 UTC m=+3855.922693344" watchObservedRunningTime="2026-07-30 20:28:53.703777129 +0000 UTC m=+3855.923336040" Jul 30 20:28:55.206554 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:55.206524 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-f2czs_52b30ff5-e40a-45b5-b6bb-80483bb83057/dns/0.log" Jul 30 20:28:55.224360 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:55.224336 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-f2czs_52b30ff5-e40a-45b5-b6bb-80483bb83057/kube-rbac-proxy/0.log" Jul 30 20:28:55.310902 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:55.310877 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5p5fm_e9a5febb-adab-42b2-a740-a057ada6294a/dns-node-resolver/0.log" Jul 30 20:28:55.821694 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:55.821664 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-bbb6865cb-vjk5x_c652215e-79e4-4fbc-a464-c6009405517c/registry/0.log" Jul 30 20:28:55.896431 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:55.896402 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rfjmx_c058e636-437f-4395-9e4a-760cc5fd051c/node-ca/0.log" Jul 30 20:28:57.009921 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:57.009888 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-6ftv2_fce6373e-9c39-45c0-9d46-a9c42a950783/discovery/0.log" Jul 30 20:28:57.028927 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:57.028907 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b4b6cc975-89d9s_b68a3551-b660-476a-84e5-66cb8328fa6b/kube-auth-proxy/0.log" Jul 30 20:28:57.148819 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:57.148785 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-w2gbm_8a9ee543-86c7-4d8e-ad53-cef7ece4bbf1/payload-pre-processing/0.log" Jul 30 20:28:57.176482 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:57.176457 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-knk59_bea701bd-a892-44cf-bedf-cbb6bd64bb37/payload-processing/0.log" Jul 30 20:28:57.993257 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:57.993230 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gng5x_335be97f-3b45-4e7c-b7d9-2a0405943696/serve-healthcheck-canary/0.log" Jul 30 20:28:58.479103 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:58.479070 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2nhzb_6145ba93-9df0-4842-9043-e63f3e67712a/kube-rbac-proxy/0.log" Jul 30 20:28:58.497593 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:58.497567 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2nhzb_6145ba93-9df0-4842-9043-e63f3e67712a/exporter/0.log" Jul 30 20:28:58.517017 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:58.516996 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2nhzb_6145ba93-9df0-4842-9043-e63f3e67712a/extractor/0.log" Jul 30 20:28:59.703140 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:28:59.703106 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fmzs6/perf-node-gather-daemonset-gr2dp" Jul 30 20:29:00.689701 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:29:00.689672 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-ccptj_86ec7077-e86c-4f03-ac13-8bded5e52536/manager/0.log" Jul 30 20:29:00.887050 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:29:00.887017 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-vrmx5_429bcbb5-b835-4e1a-9c76-f4512160f42e/manager/0.log" Jul 30 20:29:01.065253 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:29:01.065173 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-ldgsc_8684d264-ac83-4de0-90a7-da2723d55c46/server/0.log" Jul 30 20:29:01.136187 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:29:01.136159 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-ntx4s_ab7a30d6-2b2d-48f9-87d2-b773a81e2dfe/manager/0.log" Jul 30 20:29:01.342056 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:29:01.342014 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76fdfbdbf4-w5r56_0604020a-98cf-4d6c-8ee7-a12cbda5f73d/manager/0.log" Jul 30 20:29:01.357954 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:29:01.357932 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76fdfbdbf4-w5r56_0604020a-98cf-4d6c-8ee7-a12cbda5f73d/copy-manifests/0.log" Jul 30 20:29:02.794378 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:29:02.794282 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-ssx6d_3ae14bcf-646c-47b0-868a-36fb4b775e8a/openshift-lws-operator/0.log" Jul 30 20:29:03.093134 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:29:03.093047 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-6ftv2_fce6373e-9c39-45c0-9d46-a9c42a950783/discovery/0.log" Jul 30 20:29:03.103179 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:29:03.103157 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b4b6cc975-89d9s_b68a3551-b660-476a-84e5-66cb8328fa6b/kube-auth-proxy/0.log" Jul 30 20:29:03.207522 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:29:03.207489 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-w2gbm_8a9ee543-86c7-4d8e-ad53-cef7ece4bbf1/payload-pre-processing/0.log" Jul 30 20:29:03.226640 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:29:03.226612 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-knk59_bea701bd-a892-44cf-bedf-cbb6bd64bb37/payload-processing/0.log" Jul 30 20:29:04.753622 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:29:04.753542 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-6ftv2_fce6373e-9c39-45c0-9d46-a9c42a950783/discovery/0.log" Jul 30 20:29:04.766366 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:29:04.766343 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b4b6cc975-89d9s_b68a3551-b660-476a-84e5-66cb8328fa6b/kube-auth-proxy/0.log" Jul 30 20:29:04.853218 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:29:04.853198 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-w2gbm_8a9ee543-86c7-4d8e-ad53-cef7ece4bbf1/payload-pre-processing/0.log" Jul 30 20:29:04.872305 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:29:04.872283 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-knk59_bea701bd-a892-44cf-bedf-cbb6bd64bb37/payload-processing/0.log" Jul 30 20:29:09.284667 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:29:09.284638 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r8fml_6ed49186-8fa8-4875-8da3-b74f8fc2b712/kube-multus-additional-cni-plugins/0.log" Jul 30 20:29:09.302863 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:29:09.302836 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r8fml_6ed49186-8fa8-4875-8da3-b74f8fc2b712/egress-router-binary-copy/0.log" Jul 30 20:29:09.322639 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:29:09.322616 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r8fml_6ed49186-8fa8-4875-8da3-b74f8fc2b712/cni-plugins/0.log" Jul 30 20:29:09.340607 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:29:09.340587 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r8fml_6ed49186-8fa8-4875-8da3-b74f8fc2b712/bond-cni-plugin/0.log" Jul 30 20:29:09.359930 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:29:09.359910 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r8fml_6ed49186-8fa8-4875-8da3-b74f8fc2b712/routeoverride-cni/0.log" Jul 30 20:29:09.378259 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:29:09.378242 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r8fml_6ed49186-8fa8-4875-8da3-b74f8fc2b712/whereabouts-cni-bincopy/0.log" Jul 30 20:29:09.395470 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:29:09.395452 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r8fml_6ed49186-8fa8-4875-8da3-b74f8fc2b712/whereabouts-cni/0.log" Jul 30 20:29:09.428996 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:29:09.428980 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-ddrgc_47812e97-b92a-460e-8383-6728edc33f73/kube-multus/0.log" Jul 30 20:29:09.608731 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:29:09.608672 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vxwhr_9547ff50-df97-486d-85eb-c253420d0e73/network-metrics-daemon/0.log" Jul 30 20:29:09.626278 ip-10-0-129-105 kubenswrapper[2556]: I0730 20:29:09.626255 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vxwhr_9547ff50-df97-486d-85eb-c253420d0e73/kube-rbac-proxy/0.log"