Jul 10 18:58:35.810022 ip-10-0-135-67 systemd[1]: Starting Kubernetes Kubelet... Jul 10 18:58:36.336925 ip-10-0-135-67 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 18:58:36.336925 ip-10-0-135-67 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 10 18:58:36.336925 ip-10-0-135-67 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 18:58:36.336925 ip-10-0-135-67 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 10 18:58:36.336925 ip-10-0-135-67 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 18:58:36.336925 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.299843 2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 10 18:58:36.336925 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303493 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 18:58:36.336925 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303506 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 18:58:36.336925 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303509 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 18:58:36.336925 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303513 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 18:58:36.336925 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303516 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 18:58:36.336925 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303519 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 18:58:36.336925 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303522 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 18:58:36.336925 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303525 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 18:58:36.339382 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303528 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 18:58:36.339382 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303531 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 18:58:36.339382 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303533 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 18:58:36.339382 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303536 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 18:58:36.339382 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303539 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 18:58:36.339382 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303542 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 18:58:36.339382 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303545 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 18:58:36.339382 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303548 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 18:58:36.339382 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303550 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 18:58:36.339382 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303553 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 18:58:36.339382 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303557 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 18:58:36.339382 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303560 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 18:58:36.339382 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303564 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 18:58:36.339382 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303568 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 18:58:36.339382 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303572 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 18:58:36.339382 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303575 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 18:58:36.339382 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303579 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 18:58:36.339382 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303583 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 18:58:36.339382 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303586 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 18:58:36.339382 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303589 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 18:58:36.341941 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303592 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 18:58:36.341941 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303596 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 18:58:36.341941 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303599 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 18:58:36.341941 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303602 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 18:58:36.341941 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303606 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 18:58:36.341941 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303609 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 18:58:36.341941 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303612 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 18:58:36.341941 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303615 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 18:58:36.341941 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303618 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 18:58:36.341941 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303623 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 18:58:36.341941 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303632 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 18:58:36.341941 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303636 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 18:58:36.341941 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303641 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 18:58:36.341941 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303644 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 18:58:36.341941 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303648 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 18:58:36.341941 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303651 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 18:58:36.341941 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303654 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 18:58:36.341941 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303658 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 18:58:36.341941 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303661 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 18:58:36.343776 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303665 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 18:58:36.343776 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303668 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 18:58:36.343776 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303671 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 18:58:36.343776 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303678 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 18:58:36.343776 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303696 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 18:58:36.343776 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303699 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 18:58:36.343776 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303703 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 18:58:36.343776 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303706 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 18:58:36.343776 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303710 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 18:58:36.343776 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303713 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 18:58:36.343776 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303716 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 18:58:36.343776 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303736 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 18:58:36.343776 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303740 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 18:58:36.343776 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303744 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 18:58:36.343776 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303747 2581 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 18:58:36.343776 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303751 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 18:58:36.343776 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303754 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 18:58:36.343776 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303758 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 18:58:36.343776 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303761 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 18:58:36.343776 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303764 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 18:58:36.356970 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303767 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 18:58:36.356970 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303770 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 18:58:36.356970 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303774 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 18:58:36.356970 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303778 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 18:58:36.356970 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303783 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 18:58:36.356970 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303787 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 18:58:36.356970 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303790 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 18:58:36.356970 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303794 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 18:58:36.356970 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303797 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 18:58:36.356970 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303800 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 18:58:36.356970 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303805 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 18:58:36.356970 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303810 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 18:58:36.356970 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303815 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 18:58:36.356970 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303818 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 18:58:36.356970 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303822 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 18:58:36.356970 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303827 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 18:58:36.356970 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303830 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 18:58:36.356970 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303833 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 18:58:36.356970 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303837 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 18:58:36.358777 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303840 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 18:58:36.358777 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303844 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 18:58:36.358777 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303847 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 18:58:36.358777 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303850 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 18:58:36.358777 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303853 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 18:58:36.358777 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303856 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 18:58:36.358777 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303859 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 18:58:36.358777 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303863 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 18:58:36.358777 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303866 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 18:58:36.358777 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303869 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 18:58:36.358777 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303873 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 18:58:36.358777 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303876 2581 feature_gate.go:328] unrecognized feature gate: Example Jul 10 18:58:36.358777 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.303879 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 18:58:36.358777 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304653 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 18:58:36.358777 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304659 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 18:58:36.358777 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304666 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 18:58:36.358777 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304669 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 18:58:36.358777 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304672 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 18:58:36.358777 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304676 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 18:58:36.358777 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304679 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 18:58:36.361124 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304698 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 18:58:36.361124 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304701 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 18:58:36.361124 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304704 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 18:58:36.361124 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304711 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 18:58:36.361124 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304714 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 18:58:36.361124 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304717 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 18:58:36.361124 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304719 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 18:58:36.361124 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304722 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 18:58:36.361124 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304724 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 18:58:36.361124 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304727 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 18:58:36.361124 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304730 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 18:58:36.361124 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304732 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 18:58:36.361124 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304735 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 18:58:36.361124 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304738 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 18:58:36.361124 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304740 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 18:58:36.361124 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304743 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 18:58:36.361124 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304746 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 18:58:36.361124 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304748 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 18:58:36.361124 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304751 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 18:58:36.361124 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304754 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 18:58:36.372665 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304756 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 18:58:36.372665 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304759 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 18:58:36.372665 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304762 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 18:58:36.372665 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304764 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 18:58:36.372665 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304767 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 18:58:36.372665 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304769 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 18:58:36.372665 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304772 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 18:58:36.372665 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304776 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 18:58:36.372665 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304779 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 18:58:36.372665 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304781 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 18:58:36.372665 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304784 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 18:58:36.372665 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304787 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 18:58:36.372665 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304790 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 18:58:36.372665 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304794 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 18:58:36.372665 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304798 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 18:58:36.372665 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304802 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 18:58:36.372665 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304806 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 18:58:36.372665 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304809 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 18:58:36.372665 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304812 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 18:58:36.372665 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304816 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 18:58:36.375116 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304820 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 18:58:36.375116 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304824 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 18:58:36.375116 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304827 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 18:58:36.375116 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304831 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 18:58:36.375116 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304835 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 18:58:36.375116 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304839 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 18:58:36.375116 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304843 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 18:58:36.375116 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304847 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 18:58:36.375116 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304851 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 18:58:36.375116 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304856 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 18:58:36.375116 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304859 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 18:58:36.375116 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304862 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 18:58:36.375116 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304866 2581 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 18:58:36.375116 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304872 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 18:58:36.375116 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304876 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 18:58:36.375116 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304881 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 18:58:36.375116 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304885 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 18:58:36.375116 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304889 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 18:58:36.375116 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304893 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 18:58:36.375116 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304901 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 18:58:36.374727 ip-10-0-135-67 systemd[1]: Started Kubernetes Kubelet. Jul 10 18:58:36.388669 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304906 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 18:58:36.388669 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304911 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 18:58:36.388669 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304915 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 18:58:36.388669 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304920 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 18:58:36.388669 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304925 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 18:58:36.388669 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304930 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 18:58:36.388669 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304934 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 18:58:36.388669 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304938 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 18:58:36.388669 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304942 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 18:58:36.388669 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304947 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 18:58:36.388669 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304951 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 18:58:36.388669 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304955 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 18:58:36.388669 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304958 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 18:58:36.388669 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304962 2581 feature_gate.go:328] unrecognized feature gate: Example Jul 10 18:58:36.388669 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304968 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 18:58:36.388669 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304973 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 18:58:36.388669 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304978 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 18:58:36.388669 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304982 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 18:58:36.388669 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304988 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 18:58:36.405193 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304992 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 18:58:36.405193 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.304997 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 18:58:36.405193 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305002 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 18:58:36.405193 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305007 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 18:58:36.405193 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305010 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 18:58:36.405193 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305012 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 18:58:36.405193 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305015 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 18:58:36.405193 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305017 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 18:58:36.405193 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305022 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 18:58:36.405193 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305025 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 18:58:36.405193 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305027 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 18:58:36.405193 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305030 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 18:58:36.405193 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305034 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 18:58:36.405193 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305404 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 18:58:36.405193 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305408 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 18:58:36.405193 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305411 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 18:58:36.405193 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305414 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 18:58:36.405193 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305417 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 18:58:36.405193 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305422 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 18:58:36.420632 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305426 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 18:58:36.420632 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305429 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 18:58:36.420632 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305432 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 18:58:36.420632 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305435 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 18:58:36.420632 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305437 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 18:58:36.420632 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305440 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 18:58:36.420632 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305444 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 18:58:36.420632 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305448 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 18:58:36.420632 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305451 2581 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 18:58:36.420632 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305453 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 18:58:36.420632 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305455 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 18:58:36.420632 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305458 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 18:58:36.420632 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305460 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 18:58:36.420632 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305463 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 18:58:36.420632 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305466 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 18:58:36.420632 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305469 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 18:58:36.420632 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305471 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 18:58:36.420632 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305474 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 18:58:36.420632 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305476 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 18:58:36.420632 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305479 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 18:58:36.423281 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305482 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 18:58:36.423281 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305486 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 18:58:36.423281 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305489 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 18:58:36.423281 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305491 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 18:58:36.423281 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305496 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 18:58:36.423281 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305500 2581 feature_gate.go:328] unrecognized feature gate: Example Jul 10 18:58:36.423281 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305503 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 18:58:36.423281 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305506 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 18:58:36.423281 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305510 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 18:58:36.423281 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305513 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 18:58:36.423281 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305515 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 18:58:36.423281 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305519 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 18:58:36.423281 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305522 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 18:58:36.423281 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305524 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 18:58:36.423281 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305527 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 18:58:36.423281 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305529 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 18:58:36.423281 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305532 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 18:58:36.423281 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305534 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 18:58:36.423281 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305537 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 18:58:36.436760 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305539 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 18:58:36.436760 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305542 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 18:58:36.436760 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305544 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 18:58:36.436760 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305546 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 18:58:36.436760 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305549 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 18:58:36.436760 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305551 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 18:58:36.436760 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305554 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 18:58:36.436760 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305557 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 18:58:36.436760 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305559 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 18:58:36.436760 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305561 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 18:58:36.436760 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305564 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 18:58:36.436760 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305566 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 18:58:36.436760 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305569 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 18:58:36.436760 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305571 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 18:58:36.436760 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305576 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 18:58:36.436760 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305579 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 18:58:36.436760 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305583 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 18:58:36.436760 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305586 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 18:58:36.436760 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305590 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 18:58:36.454275 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305592 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 18:58:36.454275 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305595 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 18:58:36.454275 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305597 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 18:58:36.454275 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305600 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 18:58:36.454275 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305602 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 18:58:36.454275 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305605 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 18:58:36.454275 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305608 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 18:58:36.454275 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305610 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 18:58:36.454275 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305613 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 18:58:36.454275 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305615 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 18:58:36.454275 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305618 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 18:58:36.454275 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305620 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 18:58:36.454275 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305623 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 18:58:36.454275 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305625 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 18:58:36.454275 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305628 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 18:58:36.454275 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305631 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 18:58:36.454275 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305633 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 18:58:36.454275 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305636 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 18:58:36.454275 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305638 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 18:58:36.454275 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305641 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 18:58:36.607881 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305643 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 18:58:36.607881 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305646 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 18:58:36.607881 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305648 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 18:58:36.607881 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305651 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 18:58:36.607881 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305653 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 18:58:36.607881 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305656 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 18:58:36.607881 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305658 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 18:58:36.607881 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305663 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 18:58:36.607881 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305667 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 18:58:36.607881 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305670 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 18:58:36.607881 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305672 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 18:58:36.607881 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305676 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 18:58:36.607881 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305679 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 18:58:36.607881 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305698 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 18:58:36.607881 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.305703 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 18:58:36.607881 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305768 2581 flags.go:64] FLAG: --address="0.0.0.0" Jul 10 18:58:36.607881 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305774 2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 10 18:58:36.607881 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305781 2581 flags.go:64] FLAG: --anonymous-auth="true" Jul 10 18:58:36.607881 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305786 2581 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 10 18:58:36.607881 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305791 2581 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 10 18:58:36.607881 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305794 2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 10 18:58:36.610183 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305798 2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 10 18:58:36.610183 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305802 2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 10 18:58:36.610183 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305805 2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 10 18:58:36.610183 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305809 2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 10 18:58:36.610183 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305812 2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 10 18:58:36.610183 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305815 2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 10 18:58:36.610183 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305819 2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 10 18:58:36.610183 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305822 2581 flags.go:64] FLAG: --cgroup-root="" Jul 10 18:58:36.610183 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305824 2581 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 10 18:58:36.610183 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305828 2581 flags.go:64] FLAG: --client-ca-file="" Jul 10 18:58:36.610183 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305831 2581 flags.go:64] FLAG: --cloud-provider="external" Jul 10 18:58:36.610183 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305833 2581 flags.go:64] FLAG: --cluster-dns="[]" Jul 10 18:58:36.610183 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305837 2581 flags.go:64] FLAG: --cluster-domain="" Jul 10 18:58:36.610183 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305840 2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 10 18:58:36.610183 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305843 2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 10 18:58:36.610183 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305846 2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 10 18:58:36.610183 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305850 2581 flags.go:64] FLAG: --container-log-max-files="5" Jul 10 18:58:36.610183 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305854 2581 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 10 18:58:36.610183 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305859 2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 10 18:58:36.610183 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305862 2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 10 18:58:36.610183 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305865 2581 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 10 18:58:36.610183 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305868 2581 flags.go:64] FLAG: --contention-profiling="false" Jul 10 18:58:36.610183 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305872 2581 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 10 18:58:36.610183 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305875 2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 10 18:58:36.612177 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305880 2581 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 10 18:58:36.612177 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305885 2581 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 10 18:58:36.612177 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305889 2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 10 18:58:36.612177 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305892 2581 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 10 18:58:36.612177 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305894 2581 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 10 18:58:36.612177 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305897 2581 flags.go:64] FLAG: --enable-load-reader="false" Jul 10 18:58:36.612177 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305900 2581 flags.go:64] FLAG: --enable-server="true" Jul 10 18:58:36.612177 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305904 2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 10 18:58:36.612177 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305909 2581 flags.go:64] FLAG: --event-burst="100" Jul 10 18:58:36.612177 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305912 2581 flags.go:64] FLAG: --event-qps="50" Jul 10 18:58:36.612177 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305915 2581 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 10 18:58:36.612177 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305918 2581 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 10 18:58:36.612177 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305921 2581 flags.go:64] FLAG: --eviction-hard="" Jul 10 18:58:36.612177 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305925 2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 10 18:58:36.612177 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305927 2581 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 10 18:58:36.612177 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305930 2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 10 18:58:36.612177 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305933 2581 flags.go:64] FLAG: --eviction-soft="" Jul 10 18:58:36.612177 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305936 2581 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 10 18:58:36.612177 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305939 2581 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 10 18:58:36.612177 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305942 2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 10 18:58:36.612177 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305945 2581 flags.go:64] FLAG: --experimental-mounter-path="" Jul 10 18:58:36.612177 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305948 2581 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 10 18:58:36.612177 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305951 2581 flags.go:64] FLAG: --fail-swap-on="true" Jul 10 18:58:36.612177 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305954 2581 flags.go:64] FLAG: --feature-gates="" Jul 10 18:58:36.612177 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305957 2581 flags.go:64] FLAG: --file-check-frequency="20s" Jul 10 18:58:36.614808 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305960 2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 10 18:58:36.614808 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305965 2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 10 18:58:36.614808 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305968 2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 10 18:58:36.614808 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305971 2581 flags.go:64] FLAG: --healthz-port="10248" Jul 10 18:58:36.614808 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305974 2581 flags.go:64] FLAG: --help="false" Jul 10 18:58:36.614808 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305979 2581 flags.go:64] FLAG: --hostname-override="ip-10-0-135-67.ec2.internal" Jul 10 18:58:36.614808 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305982 2581 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 10 18:58:36.614808 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305985 2581 flags.go:64] FLAG: --http-check-frequency="20s" Jul 10 18:58:36.614808 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305988 2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 10 18:58:36.614808 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305991 2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 10 18:58:36.614808 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305995 2581 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 10 18:58:36.614808 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.305998 2581 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 10 18:58:36.614808 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306001 2581 flags.go:64] FLAG: --image-service-endpoint="" Jul 10 18:58:36.614808 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306004 2581 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 10 18:58:36.614808 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306007 2581 flags.go:64] FLAG: --kube-api-burst="100" Jul 10 18:58:36.614808 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306011 2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 10 18:58:36.614808 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306014 2581 flags.go:64] FLAG: --kube-api-qps="50" Jul 10 18:58:36.614808 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306017 2581 flags.go:64] FLAG: --kube-reserved="" Jul 10 18:58:36.614808 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306020 2581 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 10 18:58:36.614808 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306022 2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 10 18:58:36.614808 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306025 2581 flags.go:64] FLAG: --kubelet-cgroups="" Jul 10 18:58:36.614808 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306028 2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 10 18:58:36.614808 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306031 2581 flags.go:64] FLAG: --lock-file="" Jul 10 18:58:36.614808 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306034 2581 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 10 18:58:36.636012 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306036 2581 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 10 18:58:36.636012 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306039 2581 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 10 18:58:36.636012 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306044 2581 flags.go:64] FLAG: --log-json-split-stream="false" Jul 10 18:58:36.636012 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306047 2581 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 10 18:58:36.636012 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306050 2581 flags.go:64] FLAG: --log-text-split-stream="false" Jul 10 18:58:36.636012 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306053 2581 flags.go:64] FLAG: --logging-format="text" Jul 10 18:58:36.636012 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306055 2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 10 18:58:36.636012 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306058 2581 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 10 18:58:36.636012 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306061 2581 flags.go:64] FLAG: --manifest-url="" Jul 10 18:58:36.636012 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306065 2581 flags.go:64] FLAG: --manifest-url-header="" Jul 10 18:58:36.636012 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306069 2581 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 10 18:58:36.636012 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306072 2581 flags.go:64] FLAG: --max-open-files="1000000" Jul 10 18:58:36.636012 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306076 2581 flags.go:64] FLAG: --max-pods="110" Jul 10 18:58:36.636012 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306081 2581 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 10 18:58:36.636012 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306084 2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 10 18:58:36.636012 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306087 2581 flags.go:64] FLAG: --memory-manager-policy="None" Jul 10 18:58:36.636012 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306090 2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 10 18:58:36.636012 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306094 2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 10 18:58:36.636012 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306099 2581 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 10 18:58:36.636012 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306101 2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 10 18:58:36.636012 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306109 2581 flags.go:64] FLAG: --node-status-max-images="50" Jul 10 18:58:36.636012 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306112 2581 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 10 18:58:36.636012 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306115 2581 flags.go:64] FLAG: --oom-score-adj="-999" Jul 10 18:58:36.636012 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306118 2581 flags.go:64] FLAG: --pod-cidr="" Jul 10 18:58:36.638837 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306122 2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 10 18:58:36.638837 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306127 2581 flags.go:64] FLAG: --pod-manifest-path="" Jul 10 18:58:36.638837 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306130 2581 flags.go:64] FLAG: --pod-max-pids="-1" Jul 10 18:58:36.638837 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306134 2581 flags.go:64] FLAG: --pods-per-core="0" Jul 10 18:58:36.638837 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306136 2581 flags.go:64] FLAG: --port="10250" Jul 10 18:58:36.638837 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306139 2581 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 10 18:58:36.638837 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306142 2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ba027a850bb8025e" Jul 10 18:58:36.638837 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306145 2581 flags.go:64] FLAG: --qos-reserved="" Jul 10 18:58:36.638837 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306148 2581 flags.go:64] FLAG: --read-only-port="10255" Jul 10 18:58:36.638837 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306151 2581 flags.go:64] FLAG: --register-node="true" Jul 10 18:58:36.638837 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306154 2581 flags.go:64] FLAG: --register-with-taints="" Jul 10 18:58:36.638837 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306158 2581 flags.go:64] FLAG: --registry-burst="10" Jul 10 18:58:36.638837 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306162 2581 flags.go:64] FLAG: --registry-qps="5" Jul 10 18:58:36.638837 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306166 2581 flags.go:64] FLAG: --reserved-cpus="" Jul 10 18:58:36.638837 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306171 2581 flags.go:64] FLAG: --reserved-memory="" Jul 10 18:58:36.638837 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306177 2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 10 18:58:36.638837 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306181 2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 10 18:58:36.638837 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306187 2581 flags.go:64] FLAG: --rotate-certificates="false" Jul 10 18:58:36.638837 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306192 2581 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 10 18:58:36.638837 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306197 2581 flags.go:64] FLAG: --runonce="false" Jul 10 18:58:36.638837 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306201 2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 10 18:58:36.638837 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306205 2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 10 18:58:36.638837 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306211 2581 flags.go:64] FLAG: --seccomp-default="false" Jul 10 18:58:36.638837 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306216 2581 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 10 18:58:36.638837 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306220 2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 10 18:58:36.650304 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306225 2581 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 10 18:58:36.650304 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306230 2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 10 18:58:36.650304 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306235 2581 flags.go:64] FLAG: --storage-driver-password="root" Jul 10 18:58:36.650304 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306239 2581 flags.go:64] FLAG: --storage-driver-secure="false" Jul 10 18:58:36.650304 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306244 2581 flags.go:64] FLAG: --storage-driver-table="stats" Jul 10 18:58:36.650304 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306249 2581 flags.go:64] FLAG: --storage-driver-user="root" Jul 10 18:58:36.650304 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306253 2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 10 18:58:36.650304 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306258 2581 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 10 18:58:36.650304 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306264 2581 flags.go:64] FLAG: --system-cgroups="" Jul 10 18:58:36.650304 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306268 2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 10 18:58:36.650304 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306277 2581 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 10 18:58:36.650304 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306281 2581 flags.go:64] FLAG: --tls-cert-file="" Jul 10 18:58:36.650304 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306286 2581 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 10 18:58:36.650304 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306293 2581 flags.go:64] FLAG: --tls-min-version="" Jul 10 18:58:36.650304 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306297 2581 flags.go:64] FLAG: --tls-private-key-file="" Jul 10 18:58:36.650304 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306301 2581 flags.go:64] FLAG: --topology-manager-policy="none" Jul 10 18:58:36.650304 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306305 2581 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 10 18:58:36.650304 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306309 2581 flags.go:64] FLAG: --topology-manager-scope="container" Jul 10 18:58:36.650304 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306314 2581 flags.go:64] FLAG: --v="2" Jul 10 18:58:36.650304 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306321 2581 flags.go:64] FLAG: --version="false" Jul 10 18:58:36.650304 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306325 2581 flags.go:64] FLAG: --vmodule="" Jul 10 18:58:36.650304 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306329 2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 10 18:58:36.650304 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306333 2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 10 18:58:36.650304 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306453 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 18:58:36.650304 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306459 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 18:58:36.651092 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306463 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 18:58:36.651092 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306466 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 18:58:36.651092 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306469 2581 feature_gate.go:328] unrecognized feature gate: Example Jul 10 18:58:36.651092 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306472 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 18:58:36.651092 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306477 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 18:58:36.651092 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306480 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 18:58:36.651092 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306482 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 18:58:36.651092 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306487 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 18:58:36.651092 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306491 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 18:58:36.651092 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306494 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 18:58:36.651092 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306498 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 18:58:36.651092 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306501 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 18:58:36.651092 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306504 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 18:58:36.651092 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306507 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 18:58:36.651092 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306510 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 18:58:36.651092 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306513 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 18:58:36.651092 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306516 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 18:58:36.651092 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306519 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 18:58:36.651092 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306521 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 18:58:36.651092 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306524 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 18:58:36.651656 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306527 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 18:58:36.651656 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306529 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 18:58:36.651656 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306532 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 18:58:36.651656 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306535 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 18:58:36.651656 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306537 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 18:58:36.651656 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306540 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 18:58:36.651656 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306542 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 18:58:36.651656 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306545 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 18:58:36.651656 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306547 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 18:58:36.651656 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306550 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 18:58:36.651656 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306552 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 18:58:36.651656 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306556 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 18:58:36.651656 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306559 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 18:58:36.651656 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306561 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 18:58:36.651656 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306564 2581 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 18:58:36.651656 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306566 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 18:58:36.651656 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306570 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 18:58:36.651656 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306572 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 18:58:36.651656 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306575 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 18:58:36.651656 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306578 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 18:58:36.652262 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306580 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 18:58:36.652262 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306583 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 18:58:36.652262 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306585 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 18:58:36.652262 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306588 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 18:58:36.652262 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306590 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 18:58:36.652262 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306593 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 18:58:36.652262 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306595 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 18:58:36.652262 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306598 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 18:58:36.652262 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306600 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 18:58:36.652262 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306605 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 18:58:36.652262 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306608 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 18:58:36.652262 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306611 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 18:58:36.652262 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306614 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 18:58:36.652262 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306617 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 18:58:36.652262 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306620 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 18:58:36.652262 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306622 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 18:58:36.652262 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306625 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 18:58:36.652262 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306627 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 18:58:36.652262 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306630 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 18:58:36.652876 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306632 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 18:58:36.652876 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306635 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 18:58:36.652876 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306637 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 18:58:36.652876 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306640 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 18:58:36.652876 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306644 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 18:58:36.652876 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306647 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 18:58:36.652876 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306649 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 18:58:36.652876 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306652 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 18:58:36.652876 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306655 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 18:58:36.652876 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306659 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 18:58:36.652876 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306662 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 18:58:36.652876 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306664 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 18:58:36.652876 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306667 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 18:58:36.652876 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306669 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 18:58:36.652876 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306672 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 18:58:36.652876 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306674 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 18:58:36.652876 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306677 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 18:58:36.652876 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306679 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 18:58:36.652876 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306700 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 18:58:36.652876 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306703 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 18:58:36.653470 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306705 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 18:58:36.653470 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306708 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 18:58:36.653470 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306710 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 18:58:36.653470 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306714 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 18:58:36.653470 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306717 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 18:58:36.653470 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306720 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 18:58:36.653470 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306722 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 18:58:36.653470 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306725 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 18:58:36.653470 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306728 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 18:58:36.653470 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306730 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 18:58:36.653470 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306733 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 18:58:36.653470 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306735 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 18:58:36.653470 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306738 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 18:58:36.653470 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306740 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 18:58:36.653470 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306743 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 18:58:36.653470 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306745 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 18:58:36.653470 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306749 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 18:58:36.653470 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.306752 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 18:58:36.653968 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.306762 2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 18:58:36.653968 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.313947 2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 10 18:58:36.653968 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.313963 2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 10 18:58:36.653968 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.313982 2581 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 10 18:58:36.653968 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.313988 2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 10 18:58:36.653968 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314043 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 18:58:36.653968 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314048 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 18:58:36.653968 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314052 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 18:58:36.653968 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314055 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 18:58:36.653968 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314058 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 18:58:36.653968 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314061 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 18:58:36.653968 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314064 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 18:58:36.653968 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314067 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 18:58:36.653968 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314070 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 18:58:36.653968 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314073 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 18:58:36.654409 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314077 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 18:58:36.654409 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314080 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 18:58:36.654409 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314084 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 18:58:36.654409 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314087 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 18:58:36.654409 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314089 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 18:58:36.654409 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314092 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 18:58:36.654409 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314095 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 18:58:36.654409 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314098 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 18:58:36.654409 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314101 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 18:58:36.654409 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314103 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 18:58:36.654409 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314106 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 18:58:36.654409 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314108 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 18:58:36.654409 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314111 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 18:58:36.654409 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314114 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 18:58:36.654409 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314116 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 18:58:36.654409 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314119 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 18:58:36.654409 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314121 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 18:58:36.654409 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314124 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 18:58:36.654409 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314127 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 18:58:36.654409 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314130 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 18:58:36.655016 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314133 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 18:58:36.655016 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314135 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 18:58:36.655016 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314138 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 18:58:36.655016 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314141 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 18:58:36.655016 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314144 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 18:58:36.655016 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314147 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 18:58:36.655016 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314150 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 18:58:36.655016 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314153 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 18:58:36.655016 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314156 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 18:58:36.655016 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314158 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 18:58:36.655016 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314161 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 18:58:36.655016 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314180 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 18:58:36.655016 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314183 2581 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 18:58:36.655016 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314186 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 18:58:36.655016 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314190 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 18:58:36.655016 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314194 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 18:58:36.655016 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314197 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 18:58:36.655016 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314200 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 18:58:36.655016 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314203 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 18:58:36.655016 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314206 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 18:58:36.655980 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314209 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 18:58:36.655980 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314211 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 18:58:36.655980 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314214 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 18:58:36.655980 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314219 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 18:58:36.655980 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314223 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 18:58:36.655980 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314226 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 18:58:36.655980 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314228 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 18:58:36.655980 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314231 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 18:58:36.655980 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314235 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 18:58:36.655980 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314238 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 18:58:36.655980 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314241 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 18:58:36.655980 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314244 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 18:58:36.655980 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314247 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 18:58:36.655980 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314250 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 18:58:36.655980 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314253 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 18:58:36.655980 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314258 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 18:58:36.655980 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314261 2581 feature_gate.go:328] unrecognized feature gate: Example Jul 10 18:58:36.655980 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314264 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 18:58:36.655980 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314267 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 18:58:36.656610 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314269 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 18:58:36.656610 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314272 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 18:58:36.656610 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314275 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 18:58:36.656610 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314277 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 18:58:36.656610 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314280 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 18:58:36.656610 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314282 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 18:58:36.656610 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314286 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 18:58:36.656610 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314289 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 18:58:36.656610 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314292 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 18:58:36.656610 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314295 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 18:58:36.656610 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314298 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 18:58:36.656610 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314300 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 18:58:36.656610 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314303 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 18:58:36.656610 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314306 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 18:58:36.656610 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314308 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 18:58:36.656610 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314312 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 18:58:36.656610 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314316 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 18:58:36.656610 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314319 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 18:58:36.656610 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314321 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 18:58:36.657229 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314324 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 18:58:36.657229 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314326 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 18:58:36.657229 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314329 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 18:58:36.657229 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314331 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 18:58:36.657229 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314334 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 18:58:36.657229 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314337 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 18:58:36.657229 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314339 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 18:58:36.657229 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314342 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 18:58:36.657229 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314345 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 18:58:36.657229 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314349 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 18:58:36.657229 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314352 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 18:58:36.657229 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.314357 2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 18:58:36.657229 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314471 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 18:58:36.657229 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314476 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 18:58:36.657646 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314479 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 18:58:36.657646 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314482 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 18:58:36.657646 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314485 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 18:58:36.657646 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314487 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 18:58:36.657646 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314490 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 18:58:36.657646 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314493 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 18:58:36.657646 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314496 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 18:58:36.657646 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314499 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 18:58:36.657646 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314501 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 18:58:36.657646 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314504 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 18:58:36.657646 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314507 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 18:58:36.657646 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314510 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 18:58:36.657646 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314512 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 18:58:36.657646 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314515 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 18:58:36.657646 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314518 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 18:58:36.657646 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314521 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 18:58:36.657646 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314523 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 18:58:36.657646 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314526 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 18:58:36.657646 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314529 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 18:58:36.657646 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314531 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 18:58:36.658251 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314534 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 18:58:36.658251 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314536 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 18:58:36.658251 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314539 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 18:58:36.658251 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314542 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 18:58:36.658251 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314546 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 18:58:36.658251 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314549 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 18:58:36.658251 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314552 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 18:58:36.658251 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314555 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 18:58:36.658251 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314559 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 18:58:36.658251 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314562 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 18:58:36.658251 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314566 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 18:58:36.658251 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314569 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 18:58:36.658251 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314571 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 18:58:36.658251 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314575 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 18:58:36.658251 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314578 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 18:58:36.658251 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314582 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 18:58:36.658251 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314584 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 18:58:36.658251 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314589 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 18:58:36.658251 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314592 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 18:58:36.658865 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314594 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 18:58:36.658865 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314597 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 18:58:36.658865 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314600 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 18:58:36.658865 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314603 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 18:58:36.658865 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314605 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 18:58:36.658865 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314608 2581 feature_gate.go:328] unrecognized feature gate: Example Jul 10 18:58:36.658865 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314610 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 18:58:36.658865 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314613 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 18:58:36.658865 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314616 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 18:58:36.658865 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314619 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 18:58:36.658865 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314621 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 18:58:36.658865 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314624 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 18:58:36.658865 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314626 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 18:58:36.658865 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314629 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 18:58:36.658865 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314631 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 18:58:36.658865 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314634 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 18:58:36.658865 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314636 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 18:58:36.658865 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314639 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 18:58:36.658865 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314641 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 18:58:36.658865 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314643 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 18:58:36.659501 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314646 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 18:58:36.659501 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314650 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 18:58:36.659501 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314652 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 18:58:36.659501 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314655 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 18:58:36.659501 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314657 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 18:58:36.659501 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314660 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 18:58:36.659501 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314663 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 18:58:36.659501 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314665 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 18:58:36.659501 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314668 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 18:58:36.659501 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314671 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 18:58:36.659501 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314674 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 18:58:36.659501 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314677 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 18:58:36.659501 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314679 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 18:58:36.659501 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314697 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 18:58:36.659501 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314699 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 18:58:36.659501 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314702 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 18:58:36.659501 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314705 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 18:58:36.659501 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314708 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 18:58:36.659501 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314711 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 18:58:36.659501 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314713 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 18:58:36.660072 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314716 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 18:58:36.660072 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314718 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 18:58:36.660072 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314721 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 18:58:36.660072 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314724 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 18:58:36.660072 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314728 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 18:58:36.660072 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314731 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 18:58:36.660072 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314733 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 18:58:36.660072 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314736 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 18:58:36.660072 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314739 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 18:58:36.660072 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314742 2581 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 18:58:36.660072 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314744 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 18:58:36.660072 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314747 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 18:58:36.660072 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314749 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 18:58:36.660072 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314753 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 18:58:36.660072 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314756 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 18:58:36.660072 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314758 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 18:58:36.660072 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314761 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 18:58:36.660072 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:36.314763 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 18:58:36.660587 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.314768 2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 18:58:36.660587 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.314872 2581 server.go:962] "Client rotation is on, will bootstrap in background" Jul 10 18:58:36.660587 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.317019 2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 10 18:58:36.660587 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.318703 2581 server.go:1019] "Starting client certificate rotation" Jul 10 18:58:36.660587 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.318807 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 18:58:36.660587 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.318847 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 18:58:36.660587 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.321664 2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 18:58:36.660587 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.324897 2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 18:58:36.660587 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.337415 2581 log.go:25] "Validated CRI v1 runtime API" Jul 10 18:58:36.660587 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.340983 2581 log.go:25] "Validated CRI v1 image API" Jul 10 18:58:36.660587 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.341853 2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 10 18:58:36.660587 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.344795 2581 fs.go:135] Filesystem UUIDs: map[5351f71f-8a9f-40f8-b1d9-db44d954131d:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 97e28bcb-66bb-40aa-b455-76991e3cbf20:/dev/nvme0n1p4] Jul 10 18:58:36.660955 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.344810 2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 10 18:58:36.660955 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.350073 2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 18:58:36.866060 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.350399 2581 manager.go:216] Machine: {Timestamp:2026-07-10 18:58:36.348849144 +0000 UTC m=+0.392991636 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100525 MemoryCapacity:33164505088 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b1c6f20ecce736e23fc835dafdb81 SystemUUID:ec2b1c6f-20ec-ce73-6e23-fc835dafdb81 BootID:62991445-c835-4fab-a818-ac5681f98141 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582254592 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:6c:dd:83:15:e5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:6c:dd:83:15:e5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ae:76:07:8e:1b:aa Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164505088 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 10 18:58:36.866060 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.350494 2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 10 18:58:36.866060 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.350583 2581 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 10 18:58:36.866060 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.354309 2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 10 18:58:36.866060 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.354338 2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-67.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 10 18:58:36.866060 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.355328 2581 topology_manager.go:138] "Creating topology manager with none policy" Jul 10 18:58:36.866060 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.355340 2581 container_manager_linux.go:309] "Creating device plugin manager" Jul 10 18:58:36.866060 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.355349 2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 18:58:36.866060 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.356171 2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 18:58:36.866060 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.358061 2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 10 18:58:36.866060 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.361994 2581 state_mem.go:36] "Initialized new in-memory state store" Jul 10 18:58:36.866060 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.362108 2581 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 10 18:58:36.866060 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.362135 2581 kubelet.go:486] "Attempting to sync node with API server" Jul 10 18:58:36.866060 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.362145 2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 10 18:58:36.866060 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.362159 2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 10 18:58:36.866060 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.362170 2581 kubelet.go:398] "Adding apiserver pod source" Jul 10 18:58:36.866060 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.362180 2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 10 18:58:36.866060 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.362493 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 18:58:36.867246 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.362506 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 18:58:36.867246 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.365989 2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 10 18:58:36.867246 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.368615 2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 10 18:58:36.867246 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.368645 2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 10 18:58:36.867246 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.368845 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 10 18:58:36.867246 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.368858 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 10 18:58:36.867246 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.368873 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 10 18:58:36.867246 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.368882 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 10 18:58:36.867246 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.368892 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 10 18:58:36.867246 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.368897 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 10 18:58:36.867246 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.369539 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 10 18:58:36.867246 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.369550 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 10 18:58:36.867246 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.369559 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 10 18:58:36.867246 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.369565 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 10 18:58:36.867246 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.369582 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 10 18:58:36.867246 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.369590 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 10 18:58:36.867246 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.370711 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 10 18:58:36.867246 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.370732 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 10 18:58:36.867246 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.372257 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gmvcb" Jul 10 18:58:36.867246 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:36.372862 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 10 18:58:36.867246 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.372873 2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-67.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 10 18:58:36.867246 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:36.372918 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-67.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 10 18:58:36.867246 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.373960 2581 server.go:1268] "Started kubelet" Jul 10 18:58:36.867246 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.374282 2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 10 18:58:36.867246 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.378852 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gmvcb" Jul 10 18:58:36.867246 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.378715 2581 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 10 18:58:36.867246 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.378912 2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 10 18:58:36.867246 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.380357 2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 10 18:58:36.867246 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.382234 2581 server.go:310] "Adding debug handlers to kubelet server" Jul 10 18:58:36.867246 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:36.382342 2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 10 18:58:36.867246 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.386285 2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 10 18:58:36.867246 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.386301 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 10 18:58:36.867246 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:36.386370 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-67.ec2.internal\" not found" Jul 10 18:58:36.867246 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.387039 2581 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 10 18:58:36.867246 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.387045 2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 10 18:58:36.867246 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.387054 2581 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 10 18:58:36.867246 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.387130 2581 reconstruct.go:97] "Volume reconstruction finished" Jul 10 18:58:36.867246 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.387139 2581 reconciler.go:29] "Reconciler: start to sync state" Jul 10 18:58:36.867246 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.387194 2581 factory.go:55] Registering systemd factory Jul 10 18:58:36.867246 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.387206 2581 factory.go:223] Registration of the systemd container factory successfully Jul 10 18:58:36.867246 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.387490 2581 factory.go:153] Registering CRI-O factory Jul 10 18:58:36.867246 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.387503 2581 factory.go:223] Registration of the crio container factory successfully Jul 10 18:58:36.867246 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.387555 2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 10 18:58:36.867246 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.387606 2581 factory.go:103] Registering Raw factory Jul 10 18:58:36.867246 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.387625 2581 manager.go:1196] Started watching for new ooms in manager Jul 10 18:58:36.867246 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.388135 2581 manager.go:349] Starting recovery of all containers Jul 10 18:58:36.867246 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.395642 2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 18:58:36.867246 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.399134 2581 manager.go:354] Recovery completed Jul 10 18:58:36.869566 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.403290 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 18:58:36.869566 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.405155 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-67.ec2.internal" event="NodeHasSufficientMemory" Jul 10 18:58:36.869566 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.405187 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-67.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 18:58:36.869566 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.405201 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-67.ec2.internal" event="NodeHasSufficientPID" Jul 10 18:58:36.869566 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.406741 2581 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 10 18:58:36.869566 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.406754 2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 10 18:58:36.869566 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.406773 2581 state_mem.go:36] "Initialized new in-memory state store" Jul 10 18:58:36.869566 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.406764 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 18:58:36.869566 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.407646 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-67.ec2.internal" event="NodeHasSufficientMemory" Jul 10 18:58:36.869566 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.407701 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-67.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 18:58:36.869566 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.407733 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-67.ec2.internal" event="NodeHasSufficientPID" Jul 10 18:58:36.869566 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.408010 2581 policy_none.go:49] "None policy: Start" Jul 10 18:58:36.869566 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.408024 2581 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 10 18:58:36.869566 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.408034 2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 10 18:58:36.869566 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.409169 2581 policy_none.go:47] "Start" Jul 10 18:58:36.869566 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:36.412856 2581 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-135-67.ec2.internal\" not found" node="ip-10-0-135-67.ec2.internal" Jul 10 18:58:36.869566 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.480307 2581 manager.go:337] "Starting Device Plugin manager" Jul 10 18:58:36.869566 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:36.480348 2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 10 18:58:36.869566 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.480361 2581 server.go:87] "Starting device plugin registration server" Jul 10 18:58:36.869566 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.480659 2581 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 10 18:58:36.869566 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.480671 2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 10 18:58:36.869566 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.480765 2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 10 18:58:36.869566 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.480846 2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 10 18:58:36.869566 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.480853 2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 10 18:58:36.869566 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:36.481378 2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 10 18:58:36.869566 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:36.481418 2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-67.ec2.internal\" not found" Jul 10 18:58:36.869566 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.534038 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 10 18:58:36.869566 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.535349 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 10 18:58:36.869566 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.535398 2581 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 10 18:58:36.869566 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.535427 2581 kubelet.go:2448] "Starting kubelet main sync loop" Jul 10 18:58:36.869566 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:36.535470 2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 10 18:58:36.869566 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.538196 2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 18:58:36.869566 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.581766 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 18:58:36.869566 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.582791 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-67.ec2.internal" event="NodeHasSufficientMemory" Jul 10 18:58:36.869566 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.582820 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-67.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 18:58:36.869566 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.582833 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-67.ec2.internal" event="NodeHasSufficientPID" Jul 10 18:58:36.869566 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.582861 2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-67.ec2.internal" Jul 10 18:58:36.869566 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.593627 2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-67.ec2.internal" Jul 10 18:58:36.869566 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:36.593647 2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-67.ec2.internal\": node \"ip-10-0-135-67.ec2.internal\" not found" Jul 10 18:58:36.869566 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:36.607440 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-67.ec2.internal\" not found" Jul 10 18:58:36.869566 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.636538 2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-67.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-67.ec2.internal"] Jul 10 18:58:36.869566 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.636880 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 18:58:36.869566 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.637882 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-67.ec2.internal" event="NodeHasSufficientMemory" Jul 10 18:58:36.869566 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.637906 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-67.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 18:58:36.869566 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.637920 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-67.ec2.internal" event="NodeHasSufficientPID" Jul 10 18:58:36.869566 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.639394 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 18:58:36.870823 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.639597 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-67.ec2.internal" Jul 10 18:58:36.870823 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.639621 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 18:58:36.870823 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.640499 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-67.ec2.internal" event="NodeHasSufficientMemory" Jul 10 18:58:36.870823 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.640585 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-67.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 18:58:36.870823 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.640640 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-67.ec2.internal" event="NodeHasSufficientPID" Jul 10 18:58:36.870823 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.640984 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-67.ec2.internal" event="NodeHasSufficientMemory" Jul 10 18:58:36.870823 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.641008 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-67.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 18:58:36.870823 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.641019 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-67.ec2.internal" event="NodeHasSufficientPID" Jul 10 18:58:36.870823 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.642197 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-67.ec2.internal" Jul 10 18:58:36.870823 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.642218 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 18:58:36.870823 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.642805 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-67.ec2.internal" event="NodeHasSufficientMemory" Jul 10 18:58:36.870823 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.642825 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-67.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 18:58:36.870823 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.642835 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-67.ec2.internal" event="NodeHasSufficientPID" Jul 10 18:58:36.870823 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:36.663271 2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-67.ec2.internal\" not found" node="ip-10-0-135-67.ec2.internal" Jul 10 18:58:36.870823 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:36.667190 2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-67.ec2.internal\" not found" node="ip-10-0-135-67.ec2.internal" Jul 10 18:58:36.870823 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.691295 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a2521e690c042ce0e26fc7818bf7a692-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-67.ec2.internal\" (UID: \"a2521e690c042ce0e26fc7818bf7a692\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-67.ec2.internal" Jul 10 18:58:36.870823 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.691319 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2521e690c042ce0e26fc7818bf7a692-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-67.ec2.internal\" (UID: \"a2521e690c042ce0e26fc7818bf7a692\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-67.ec2.internal" Jul 10 18:58:36.870823 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.691337 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e2729654f7488332f5966486449c5669-config\") pod \"kube-apiserver-proxy-ip-10-0-135-67.ec2.internal\" (UID: \"e2729654f7488332f5966486449c5669\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-67.ec2.internal" Jul 10 18:58:36.870823 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:36.708399 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-67.ec2.internal\" not found" Jul 10 18:58:36.870823 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.791612 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e2729654f7488332f5966486449c5669-config\") pod \"kube-apiserver-proxy-ip-10-0-135-67.ec2.internal\" (UID: \"e2729654f7488332f5966486449c5669\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-67.ec2.internal" Jul 10 18:58:36.870823 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.791554 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e2729654f7488332f5966486449c5669-config\") pod \"kube-apiserver-proxy-ip-10-0-135-67.ec2.internal\" (UID: \"e2729654f7488332f5966486449c5669\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-67.ec2.internal" Jul 10 18:58:36.870823 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.791674 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a2521e690c042ce0e26fc7818bf7a692-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-67.ec2.internal\" (UID: \"a2521e690c042ce0e26fc7818bf7a692\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-67.ec2.internal" Jul 10 18:58:36.870823 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.791711 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2521e690c042ce0e26fc7818bf7a692-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-67.ec2.internal\" (UID: \"a2521e690c042ce0e26fc7818bf7a692\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-67.ec2.internal" Jul 10 18:58:36.870823 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.791739 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2521e690c042ce0e26fc7818bf7a692-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-67.ec2.internal\" (UID: \"a2521e690c042ce0e26fc7818bf7a692\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-67.ec2.internal" Jul 10 18:58:36.871473 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.791769 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a2521e690c042ce0e26fc7818bf7a692-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-67.ec2.internal\" (UID: \"a2521e690c042ce0e26fc7818bf7a692\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-67.ec2.internal" Jul 10 18:58:36.871473 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:36.809047 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-67.ec2.internal\" not found" Jul 10 18:58:37.043229 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:36.909665 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-67.ec2.internal\" not found" Jul 10 18:58:37.043229 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.965858 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-67.ec2.internal" Jul 10 18:58:37.043229 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:36.970319 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-67.ec2.internal" Jul 10 18:58:37.043229 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:37.010641 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-67.ec2.internal\" not found" Jul 10 18:58:37.189331 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:37.112115 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-67.ec2.internal\" not found" Jul 10 18:58:37.360734 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:37.213334 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-67.ec2.internal\" not found" Jul 10 18:58:37.360734 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.310138 2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 18:58:37.360734 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.317758 2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 10 18:58:37.360734 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.317890 2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 18:58:37.360734 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.317927 2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 18:58:37.360734 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.317943 2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 18:58:37.469569 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.362508 2581 apiserver.go:52] "Watching apiserver" Jul 10 18:58:37.469569 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.381484 2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 10 18:58:37.469569 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.381578 2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 18:58:37.469569 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.382987 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-zlj4w","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l298r","openshift-cluster-node-tuning-operator/tuned-k6l5w","openshift-dns/node-resolver-zf82d","openshift-multus/network-metrics-daemon-brfp2","openshift-network-operator/iptables-alerter-5z5v5","openshift-ovn-kubernetes/ovnkube-node-69hnr","kube-system/global-pull-secret-syncer-qc4kd","openshift-image-registry/node-ca-7btwf","openshift-multus/multus-additional-cni-plugins-zxs79","openshift-multus/multus-gjmqz","openshift-network-diagnostics/network-check-target-4fmmt"] Jul 10 18:58:37.469569 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.384283 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-09 18:53:37 +0000 UTC" deadline="2028-02-04 17:49:52.127312499 +0000 UTC" Jul 10 18:58:37.469569 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.384303 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13774h51m14.743011695s" Jul 10 18:58:37.469569 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.384409 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zlj4w" Jul 10 18:58:37.469569 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.386862 2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-67.ec2.internal" Jul 10 18:58:37.469569 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.386992 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 10 18:58:37.469569 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.387200 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l298r" Jul 10 18:58:37.469569 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.388470 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-csww5\"" Jul 10 18:58:37.469569 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.388659 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 10 18:58:37.469569 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.388986 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 10 18:58:37.469569 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.390685 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-k6l5w" Jul 10 18:58:37.469569 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.392472 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zf82d" Jul 10 18:58:37.469569 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.392494 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 10 18:58:37.469569 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.392997 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 10 18:58:37.469569 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.393025 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 10 18:58:37.469569 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.393478 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8g9sm\"" Jul 10 18:58:37.469569 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.393755 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-brfp2" Jul 10 18:58:37.469569 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:37.393838 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-brfp2" podUID="71f50128-1d81-4578-8758-cc4b92443331" Jul 10 18:58:37.469569 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.393851 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 10 18:58:37.469569 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.394122 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4hndr\"" Jul 10 18:58:37.469569 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.395077 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/77ed8869-fdfb-453b-b2ca-c8268fd0d0aa-device-dir\") pod \"aws-ebs-csi-driver-node-l298r\" (UID: \"77ed8869-fdfb-453b-b2ca-c8268fd0d0aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l298r" Jul 10 18:58:37.469569 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.395103 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/77ed8869-fdfb-453b-b2ca-c8268fd0d0aa-etc-selinux\") pod \"aws-ebs-csi-driver-node-l298r\" (UID: \"77ed8869-fdfb-453b-b2ca-c8268fd0d0aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l298r" Jul 10 18:58:37.469569 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.395122 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/77ed8869-fdfb-453b-b2ca-c8268fd0d0aa-sys-fs\") pod \"aws-ebs-csi-driver-node-l298r\" (UID: \"77ed8869-fdfb-453b-b2ca-c8268fd0d0aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l298r" Jul 10 18:58:37.469569 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.395145 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpgzt\" (UniqueName: \"kubernetes.io/projected/77ed8869-fdfb-453b-b2ca-c8268fd0d0aa-kube-api-access-zpgzt\") pod \"aws-ebs-csi-driver-node-l298r\" (UID: \"77ed8869-fdfb-453b-b2ca-c8268fd0d0aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l298r" Jul 10 18:58:37.471231 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.395174 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2c842677-9d8d-4210-9a3b-ff32bf8bda1e-agent-certs\") pod \"konnectivity-agent-zlj4w\" (UID: \"2c842677-9d8d-4210-9a3b-ff32bf8bda1e\") " pod="kube-system/konnectivity-agent-zlj4w" Jul 10 18:58:37.471231 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.395148 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 10 18:58:37.471231 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.395199 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2c842677-9d8d-4210-9a3b-ff32bf8bda1e-konnectivity-ca\") pod \"konnectivity-agent-zlj4w\" (UID: \"2c842677-9d8d-4210-9a3b-ff32bf8bda1e\") " pod="kube-system/konnectivity-agent-zlj4w" Jul 10 18:58:37.471231 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.395209 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 10 18:58:37.471231 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.395223 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/77ed8869-fdfb-453b-b2ca-c8268fd0d0aa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l298r\" (UID: \"77ed8869-fdfb-453b-b2ca-c8268fd0d0aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l298r" Jul 10 18:58:37.471231 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.395244 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-g4kz7\"" Jul 10 18:58:37.471231 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.395248 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/77ed8869-fdfb-453b-b2ca-c8268fd0d0aa-socket-dir\") pod \"aws-ebs-csi-driver-node-l298r\" (UID: \"77ed8869-fdfb-453b-b2ca-c8268fd0d0aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l298r" Jul 10 18:58:37.471231 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.395274 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 10 18:58:37.471231 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.395274 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/77ed8869-fdfb-453b-b2ca-c8268fd0d0aa-registration-dir\") pod \"aws-ebs-csi-driver-node-l298r\" (UID: \"77ed8869-fdfb-453b-b2ca-c8268fd0d0aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l298r" Jul 10 18:58:37.471231 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.395475 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5z5v5" Jul 10 18:58:37.471231 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.397212 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.471231 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.398704 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qc4kd" Jul 10 18:58:37.471231 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:37.398761 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qc4kd" podUID="f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe" Jul 10 18:58:37.471231 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.400106 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7btwf" Jul 10 18:58:37.471231 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.400886 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 10 18:58:37.471231 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.400894 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 10 18:58:37.471231 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.401239 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 10 18:58:37.471231 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.401681 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zxs79" Jul 10 18:58:37.471231 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.402155 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 10 18:58:37.471231 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.402378 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 10 18:58:37.471231 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.402520 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-gjr6d\"" Jul 10 18:58:37.471231 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.402543 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 10 18:58:37.471231 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.402635 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 10 18:58:37.471231 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.402660 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 10 18:58:37.471231 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.402922 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 10 18:58:37.471231 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.402952 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tlv89\"" Jul 10 18:58:37.471231 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.402978 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 10 18:58:37.471231 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.403165 2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 18:58:37.471231 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.403214 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 10 18:58:37.472661 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.403224 2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-67.ec2.internal" Jul 10 18:58:37.472661 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.403266 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lkv45\"" Jul 10 18:58:37.472661 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.403472 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 10 18:58:37.472661 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.403572 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gjmqz" Jul 10 18:58:37.472661 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.405476 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-67.ec2.internal"] Jul 10 18:58:37.472661 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.405530 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 10 18:58:37.472661 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.405556 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4fmmt" Jul 10 18:58:37.472661 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.405600 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 10 18:58:37.472661 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:37.405602 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4fmmt" podUID="48523b16-8cba-40d8-bbaf-e05814fcb820" Jul 10 18:58:37.472661 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.405631 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 10 18:58:37.472661 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.406974 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 10 18:58:37.472661 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.407048 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mv4ln\"" Jul 10 18:58:37.472661 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.407099 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 10 18:58:37.472661 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.407190 2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 18:58:37.472661 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.407442 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 10 18:58:37.472661 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.407617 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-jtl22\"" Jul 10 18:58:37.472661 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.417117 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-135-67.ec2.internal"] Jul 10 18:58:37.472661 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.417357 2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 18:58:37.472661 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.431065 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5xmnd" Jul 10 18:58:37.472661 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.440215 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5xmnd" Jul 10 18:58:37.580840 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.487980 2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 10 18:58:37.580840 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.496550 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d16a97b0-104d-435b-95b9-24f19a6dbffb-host-run-k8s-cni-cncf-io\") pod \"multus-gjmqz\" (UID: \"d16a97b0-104d-435b-95b9-24f19a6dbffb\") " pod="openshift-multus/multus-gjmqz" Jul 10 18:58:37.580840 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.496585 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d16a97b0-104d-435b-95b9-24f19a6dbffb-host-var-lib-cni-multus\") pod \"multus-gjmqz\" (UID: \"d16a97b0-104d-435b-95b9-24f19a6dbffb\") " pod="openshift-multus/multus-gjmqz" Jul 10 18:58:37.580840 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.496610 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/77ed8869-fdfb-453b-b2ca-c8268fd0d0aa-sys-fs\") pod \"aws-ebs-csi-driver-node-l298r\" (UID: \"77ed8869-fdfb-453b-b2ca-c8268fd0d0aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l298r" Jul 10 18:58:37.580840 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.496629 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8cb72f0a-5a49-4633-8ca4-3a4e240a4aab-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zxs79\" (UID: \"8cb72f0a-5a49-4633-8ca4-3a4e240a4aab\") " pod="openshift-multus/multus-additional-cni-plugins-zxs79" Jul 10 18:58:37.580840 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.496654 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvsjx\" (UniqueName: \"kubernetes.io/projected/8cb72f0a-5a49-4633-8ca4-3a4e240a4aab-kube-api-access-pvsjx\") pod \"multus-additional-cni-plugins-zxs79\" (UID: \"8cb72f0a-5a49-4633-8ca4-3a4e240a4aab\") " pod="openshift-multus/multus-additional-cni-plugins-zxs79" Jul 10 18:58:37.580840 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.496702 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d16a97b0-104d-435b-95b9-24f19a6dbffb-multus-cni-dir\") pod \"multus-gjmqz\" (UID: \"d16a97b0-104d-435b-95b9-24f19a6dbffb\") " pod="openshift-multus/multus-gjmqz" Jul 10 18:58:37.580840 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.496740 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d16a97b0-104d-435b-95b9-24f19a6dbffb-os-release\") pod \"multus-gjmqz\" (UID: \"d16a97b0-104d-435b-95b9-24f19a6dbffb\") " pod="openshift-multus/multus-gjmqz" Jul 10 18:58:37.580840 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.496768 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d16a97b0-104d-435b-95b9-24f19a6dbffb-multus-daemon-config\") pod \"multus-gjmqz\" (UID: \"d16a97b0-104d-435b-95b9-24f19a6dbffb\") " pod="openshift-multus/multus-gjmqz" Jul 10 18:58:37.580840 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.496761 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/77ed8869-fdfb-453b-b2ca-c8268fd0d0aa-sys-fs\") pod \"aws-ebs-csi-driver-node-l298r\" (UID: \"77ed8869-fdfb-453b-b2ca-c8268fd0d0aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l298r" Jul 10 18:58:37.580840 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.496810 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c807635a-eda8-4c7d-84bb-2f9da8c355ea-etc-sysconfig\") pod \"tuned-k6l5w\" (UID: \"c807635a-eda8-4c7d-84bb-2f9da8c355ea\") " pod="openshift-cluster-node-tuning-operator/tuned-k6l5w" Jul 10 18:58:37.580840 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.496831 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c807635a-eda8-4c7d-84bb-2f9da8c355ea-etc-kubernetes\") pod \"tuned-k6l5w\" (UID: \"c807635a-eda8-4c7d-84bb-2f9da8c355ea\") " pod="openshift-cluster-node-tuning-operator/tuned-k6l5w" Jul 10 18:58:37.580840 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.496847 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c807635a-eda8-4c7d-84bb-2f9da8c355ea-sys\") pod \"tuned-k6l5w\" (UID: \"c807635a-eda8-4c7d-84bb-2f9da8c355ea\") " pod="openshift-cluster-node-tuning-operator/tuned-k6l5w" Jul 10 18:58:37.580840 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.496866 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c807635a-eda8-4c7d-84bb-2f9da8c355ea-host\") pod \"tuned-k6l5w\" (UID: \"c807635a-eda8-4c7d-84bb-2f9da8c355ea\") " pod="openshift-cluster-node-tuning-operator/tuned-k6l5w" Jul 10 18:58:37.580840 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.496883 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d16a97b0-104d-435b-95b9-24f19a6dbffb-host-run-netns\") pod \"multus-gjmqz\" (UID: \"d16a97b0-104d-435b-95b9-24f19a6dbffb\") " pod="openshift-multus/multus-gjmqz" Jul 10 18:58:37.580840 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.496900 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c807635a-eda8-4c7d-84bb-2f9da8c355ea-etc-modprobe-d\") pod \"tuned-k6l5w\" (UID: \"c807635a-eda8-4c7d-84bb-2f9da8c355ea\") " pod="openshift-cluster-node-tuning-operator/tuned-k6l5w" Jul 10 18:58:37.580840 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.496929 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzkpw\" (UniqueName: \"kubernetes.io/projected/df38c8b2-26d6-46bb-95ca-166ff4723ceb-kube-api-access-xzkpw\") pod \"node-resolver-zf82d\" (UID: \"df38c8b2-26d6-46bb-95ca-166ff4723ceb\") " pod="openshift-dns/node-resolver-zf82d" Jul 10 18:58:37.581927 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.496965 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7814607d-ae88-4915-bc20-345f93f50914-iptables-alerter-script\") pod \"iptables-alerter-5z5v5\" (UID: \"7814607d-ae88-4915-bc20-345f93f50914\") " pod="openshift-network-operator/iptables-alerter-5z5v5" Jul 10 18:58:37.581927 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.496984 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsq2d\" (UniqueName: \"kubernetes.io/projected/7814607d-ae88-4915-bc20-345f93f50914-kube-api-access-vsq2d\") pod \"iptables-alerter-5z5v5\" (UID: \"7814607d-ae88-4915-bc20-345f93f50914\") " pod="openshift-network-operator/iptables-alerter-5z5v5" Jul 10 18:58:37.581927 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.497002 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7f479cec-2fb8-45c4-ae7d-073194a5dede-host-run-netns\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.581927 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.497030 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2c842677-9d8d-4210-9a3b-ff32bf8bda1e-konnectivity-ca\") pod \"konnectivity-agent-zlj4w\" (UID: \"2c842677-9d8d-4210-9a3b-ff32bf8bda1e\") " pod="kube-system/konnectivity-agent-zlj4w" Jul 10 18:58:37.581927 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.497046 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/77ed8869-fdfb-453b-b2ca-c8268fd0d0aa-device-dir\") pod \"aws-ebs-csi-driver-node-l298r\" (UID: \"77ed8869-fdfb-453b-b2ca-c8268fd0d0aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l298r" Jul 10 18:58:37.581927 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.497068 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d16a97b0-104d-435b-95b9-24f19a6dbffb-multus-conf-dir\") pod \"multus-gjmqz\" (UID: \"d16a97b0-104d-435b-95b9-24f19a6dbffb\") " pod="openshift-multus/multus-gjmqz" Jul 10 18:58:37.581927 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.497093 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c807635a-eda8-4c7d-84bb-2f9da8c355ea-etc-tuned\") pod \"tuned-k6l5w\" (UID: \"c807635a-eda8-4c7d-84bb-2f9da8c355ea\") " pod="openshift-cluster-node-tuning-operator/tuned-k6l5w" Jul 10 18:58:37.581927 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.497102 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/77ed8869-fdfb-453b-b2ca-c8268fd0d0aa-device-dir\") pod \"aws-ebs-csi-driver-node-l298r\" (UID: \"77ed8869-fdfb-453b-b2ca-c8268fd0d0aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l298r" Jul 10 18:58:37.581927 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.497110 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f479cec-2fb8-45c4-ae7d-073194a5dede-host-run-ovn-kubernetes\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.581927 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.497166 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f479cec-2fb8-45c4-ae7d-073194a5dede-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.581927 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.497185 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7f479cec-2fb8-45c4-ae7d-073194a5dede-ovnkube-config\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.581927 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.497200 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7f479cec-2fb8-45c4-ae7d-073194a5dede-ovn-node-metrics-cert\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.581927 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.497224 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe-kubelet-config\") pod \"global-pull-secret-syncer-qc4kd\" (UID: \"f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe\") " pod="kube-system/global-pull-secret-syncer-qc4kd" Jul 10 18:58:37.581927 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.497242 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8cb72f0a-5a49-4633-8ca4-3a4e240a4aab-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zxs79\" (UID: \"8cb72f0a-5a49-4633-8ca4-3a4e240a4aab\") " pod="openshift-multus/multus-additional-cni-plugins-zxs79" Jul 10 18:58:37.581927 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.497269 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f479cec-2fb8-45c4-ae7d-073194a5dede-etc-openvswitch\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.581927 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.497297 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxpbj\" (UniqueName: \"kubernetes.io/projected/7f479cec-2fb8-45c4-ae7d-073194a5dede-kube-api-access-fxpbj\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.582875 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.497318 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpgzt\" (UniqueName: \"kubernetes.io/projected/77ed8869-fdfb-453b-b2ca-c8268fd0d0aa-kube-api-access-zpgzt\") pod \"aws-ebs-csi-driver-node-l298r\" (UID: \"77ed8869-fdfb-453b-b2ca-c8268fd0d0aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l298r" Jul 10 18:58:37.582875 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.497333 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d16a97b0-104d-435b-95b9-24f19a6dbffb-system-cni-dir\") pod \"multus-gjmqz\" (UID: \"d16a97b0-104d-435b-95b9-24f19a6dbffb\") " pod="openshift-multus/multus-gjmqz" Jul 10 18:58:37.582875 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.497359 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d16a97b0-104d-435b-95b9-24f19a6dbffb-host-var-lib-kubelet\") pod \"multus-gjmqz\" (UID: \"d16a97b0-104d-435b-95b9-24f19a6dbffb\") " pod="openshift-multus/multus-gjmqz" Jul 10 18:58:37.582875 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.497392 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwxth\" (UniqueName: \"kubernetes.io/projected/48523b16-8cba-40d8-bbaf-e05814fcb820-kube-api-access-vwxth\") pod \"network-check-target-4fmmt\" (UID: \"48523b16-8cba-40d8-bbaf-e05814fcb820\") " pod="openshift-network-diagnostics/network-check-target-4fmmt" Jul 10 18:58:37.582875 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.497428 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7814607d-ae88-4915-bc20-345f93f50914-host-slash\") pod \"iptables-alerter-5z5v5\" (UID: \"7814607d-ae88-4915-bc20-345f93f50914\") " pod="openshift-network-operator/iptables-alerter-5z5v5" Jul 10 18:58:37.582875 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.497456 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7f479cec-2fb8-45c4-ae7d-073194a5dede-log-socket\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.582875 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.497488 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2c842677-9d8d-4210-9a3b-ff32bf8bda1e-agent-certs\") pod \"konnectivity-agent-zlj4w\" (UID: \"2c842677-9d8d-4210-9a3b-ff32bf8bda1e\") " pod="kube-system/konnectivity-agent-zlj4w" Jul 10 18:58:37.582875 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.497549 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5l2l\" (UniqueName: \"kubernetes.io/projected/10a5a151-ab9d-45db-8502-26d00ec21659-kube-api-access-f5l2l\") pod \"node-ca-7btwf\" (UID: \"10a5a151-ab9d-45db-8502-26d00ec21659\") " pod="openshift-image-registry/node-ca-7btwf" Jul 10 18:58:37.582875 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.497589 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2c842677-9d8d-4210-9a3b-ff32bf8bda1e-konnectivity-ca\") pod \"konnectivity-agent-zlj4w\" (UID: \"2c842677-9d8d-4210-9a3b-ff32bf8bda1e\") " pod="kube-system/konnectivity-agent-zlj4w" Jul 10 18:58:37.582875 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.497594 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d16a97b0-104d-435b-95b9-24f19a6dbffb-host-var-lib-cni-bin\") pod \"multus-gjmqz\" (UID: \"d16a97b0-104d-435b-95b9-24f19a6dbffb\") " pod="openshift-multus/multus-gjmqz" Jul 10 18:58:37.582875 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.497624 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c807635a-eda8-4c7d-84bb-2f9da8c355ea-etc-systemd\") pod \"tuned-k6l5w\" (UID: \"c807635a-eda8-4c7d-84bb-2f9da8c355ea\") " pod="openshift-cluster-node-tuning-operator/tuned-k6l5w" Jul 10 18:58:37.582875 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.497650 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c807635a-eda8-4c7d-84bb-2f9da8c355ea-lib-modules\") pod \"tuned-k6l5w\" (UID: \"c807635a-eda8-4c7d-84bb-2f9da8c355ea\") " pod="openshift-cluster-node-tuning-operator/tuned-k6l5w" Jul 10 18:58:37.582875 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.497672 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/df38c8b2-26d6-46bb-95ca-166ff4723ceb-hosts-file\") pod \"node-resolver-zf82d\" (UID: \"df38c8b2-26d6-46bb-95ca-166ff4723ceb\") " pod="openshift-dns/node-resolver-zf82d" Jul 10 18:58:37.582875 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.497687 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7f479cec-2fb8-45c4-ae7d-073194a5dede-host-slash\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.582875 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.497720 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/10a5a151-ab9d-45db-8502-26d00ec21659-host\") pod \"node-ca-7btwf\" (UID: \"10a5a151-ab9d-45db-8502-26d00ec21659\") " pod="openshift-image-registry/node-ca-7btwf" Jul 10 18:58:37.582875 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.497774 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8cb72f0a-5a49-4633-8ca4-3a4e240a4aab-system-cni-dir\") pod \"multus-additional-cni-plugins-zxs79\" (UID: \"8cb72f0a-5a49-4633-8ca4-3a4e240a4aab\") " pod="openshift-multus/multus-additional-cni-plugins-zxs79" Jul 10 18:58:37.582875 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.497792 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7f479cec-2fb8-45c4-ae7d-073194a5dede-run-ovn\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.584044 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.497811 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/77ed8869-fdfb-453b-b2ca-c8268fd0d0aa-socket-dir\") pod \"aws-ebs-csi-driver-node-l298r\" (UID: \"77ed8869-fdfb-453b-b2ca-c8268fd0d0aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l298r" Jul 10 18:58:37.584044 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.497828 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/77ed8869-fdfb-453b-b2ca-c8268fd0d0aa-etc-selinux\") pod \"aws-ebs-csi-driver-node-l298r\" (UID: \"77ed8869-fdfb-453b-b2ca-c8268fd0d0aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l298r" Jul 10 18:58:37.584044 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.497844 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/10a5a151-ab9d-45db-8502-26d00ec21659-serviceca\") pod \"node-ca-7btwf\" (UID: \"10a5a151-ab9d-45db-8502-26d00ec21659\") " pod="openshift-image-registry/node-ca-7btwf" Jul 10 18:58:37.584044 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.497876 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8cb72f0a-5a49-4633-8ca4-3a4e240a4aab-cnibin\") pod \"multus-additional-cni-plugins-zxs79\" (UID: \"8cb72f0a-5a49-4633-8ca4-3a4e240a4aab\") " pod="openshift-multus/multus-additional-cni-plugins-zxs79" Jul 10 18:58:37.584044 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.497884 2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 10 18:58:37.584044 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.497899 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8cb72f0a-5a49-4633-8ca4-3a4e240a4aab-os-release\") pod \"multus-additional-cni-plugins-zxs79\" (UID: \"8cb72f0a-5a49-4633-8ca4-3a4e240a4aab\") " pod="openshift-multus/multus-additional-cni-plugins-zxs79" Jul 10 18:58:37.584044 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.497917 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d16a97b0-104d-435b-95b9-24f19a6dbffb-cni-binary-copy\") pod \"multus-gjmqz\" (UID: \"d16a97b0-104d-435b-95b9-24f19a6dbffb\") " pod="openshift-multus/multus-gjmqz" Jul 10 18:58:37.584044 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.497931 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/77ed8869-fdfb-453b-b2ca-c8268fd0d0aa-etc-selinux\") pod \"aws-ebs-csi-driver-node-l298r\" (UID: \"77ed8869-fdfb-453b-b2ca-c8268fd0d0aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l298r" Jul 10 18:58:37.584044 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.497938 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7f479cec-2fb8-45c4-ae7d-073194a5dede-run-systemd\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.584044 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.497963 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f479cec-2fb8-45c4-ae7d-073194a5dede-run-openvswitch\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.584044 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.497961 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/77ed8869-fdfb-453b-b2ca-c8268fd0d0aa-socket-dir\") pod \"aws-ebs-csi-driver-node-l298r\" (UID: \"77ed8869-fdfb-453b-b2ca-c8268fd0d0aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l298r" Jul 10 18:58:37.584044 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.498000 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe-dbus\") pod \"global-pull-secret-syncer-qc4kd\" (UID: \"f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe\") " pod="kube-system/global-pull-secret-syncer-qc4kd" Jul 10 18:58:37.584044 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.498029 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d16a97b0-104d-435b-95b9-24f19a6dbffb-multus-socket-dir-parent\") pod \"multus-gjmqz\" (UID: \"d16a97b0-104d-435b-95b9-24f19a6dbffb\") " pod="openshift-multus/multus-gjmqz" Jul 10 18:58:37.584044 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.498044 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7f479cec-2fb8-45c4-ae7d-073194a5dede-node-log\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.584044 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.498058 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7f479cec-2fb8-45c4-ae7d-073194a5dede-env-overrides\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.584044 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.498075 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8cb72f0a-5a49-4633-8ca4-3a4e240a4aab-cni-binary-copy\") pod \"multus-additional-cni-plugins-zxs79\" (UID: \"8cb72f0a-5a49-4633-8ca4-3a4e240a4aab\") " pod="openshift-multus/multus-additional-cni-plugins-zxs79" Jul 10 18:58:37.584044 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.498099 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d16a97b0-104d-435b-95b9-24f19a6dbffb-hostroot\") pod \"multus-gjmqz\" (UID: \"d16a97b0-104d-435b-95b9-24f19a6dbffb\") " pod="openshift-multus/multus-gjmqz" Jul 10 18:58:37.585238 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.498137 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d16a97b0-104d-435b-95b9-24f19a6dbffb-etc-kubernetes\") pod \"multus-gjmqz\" (UID: \"d16a97b0-104d-435b-95b9-24f19a6dbffb\") " pod="openshift-multus/multus-gjmqz" Jul 10 18:58:37.585238 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.498175 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c807635a-eda8-4c7d-84bb-2f9da8c355ea-run\") pod \"tuned-k6l5w\" (UID: \"c807635a-eda8-4c7d-84bb-2f9da8c355ea\") " pod="openshift-cluster-node-tuning-operator/tuned-k6l5w" Jul 10 18:58:37.585238 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.498200 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c807635a-eda8-4c7d-84bb-2f9da8c355ea-tmp\") pod \"tuned-k6l5w\" (UID: \"c807635a-eda8-4c7d-84bb-2f9da8c355ea\") " pod="openshift-cluster-node-tuning-operator/tuned-k6l5w" Jul 10 18:58:37.585238 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.498217 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7t8r\" (UniqueName: \"kubernetes.io/projected/71f50128-1d81-4578-8758-cc4b92443331-kube-api-access-g7t8r\") pod \"network-metrics-daemon-brfp2\" (UID: \"71f50128-1d81-4578-8758-cc4b92443331\") " pod="openshift-multus/network-metrics-daemon-brfp2" Jul 10 18:58:37.585238 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.498232 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c807635a-eda8-4c7d-84bb-2f9da8c355ea-etc-sysctl-d\") pod \"tuned-k6l5w\" (UID: \"c807635a-eda8-4c7d-84bb-2f9da8c355ea\") " pod="openshift-cluster-node-tuning-operator/tuned-k6l5w" Jul 10 18:58:37.585238 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.498248 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7f479cec-2fb8-45c4-ae7d-073194a5dede-systemd-units\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.585238 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.498273 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f479cec-2fb8-45c4-ae7d-073194a5dede-var-lib-openvswitch\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.585238 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.498297 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/77ed8869-fdfb-453b-b2ca-c8268fd0d0aa-registration-dir\") pod \"aws-ebs-csi-driver-node-l298r\" (UID: \"77ed8869-fdfb-453b-b2ca-c8268fd0d0aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l298r" Jul 10 18:58:37.585238 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.498313 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8cb72f0a-5a49-4633-8ca4-3a4e240a4aab-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zxs79\" (UID: \"8cb72f0a-5a49-4633-8ca4-3a4e240a4aab\") " pod="openshift-multus/multus-additional-cni-plugins-zxs79" Jul 10 18:58:37.585238 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.498333 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d16a97b0-104d-435b-95b9-24f19a6dbffb-host-run-multus-certs\") pod \"multus-gjmqz\" (UID: \"d16a97b0-104d-435b-95b9-24f19a6dbffb\") " pod="openshift-multus/multus-gjmqz" Jul 10 18:58:37.585238 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.498353 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk2tf\" (UniqueName: \"kubernetes.io/projected/d16a97b0-104d-435b-95b9-24f19a6dbffb-kube-api-access-vk2tf\") pod \"multus-gjmqz\" (UID: \"d16a97b0-104d-435b-95b9-24f19a6dbffb\") " pod="openshift-multus/multus-gjmqz" Jul 10 18:58:37.585238 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.498368 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c807635a-eda8-4c7d-84bb-2f9da8c355ea-var-lib-kubelet\") pod \"tuned-k6l5w\" (UID: \"c807635a-eda8-4c7d-84bb-2f9da8c355ea\") " pod="openshift-cluster-node-tuning-operator/tuned-k6l5w" Jul 10 18:58:37.585238 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.498368 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/77ed8869-fdfb-453b-b2ca-c8268fd0d0aa-registration-dir\") pod \"aws-ebs-csi-driver-node-l298r\" (UID: \"77ed8869-fdfb-453b-b2ca-c8268fd0d0aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l298r" Jul 10 18:58:37.585238 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.498383 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df38c8b2-26d6-46bb-95ca-166ff4723ceb-tmp-dir\") pod \"node-resolver-zf82d\" (UID: \"df38c8b2-26d6-46bb-95ca-166ff4723ceb\") " pod="openshift-dns/node-resolver-zf82d" Jul 10 18:58:37.585238 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.498403 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f479cec-2fb8-45c4-ae7d-073194a5dede-host-kubelet\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.585238 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.498421 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d16a97b0-104d-435b-95b9-24f19a6dbffb-cnibin\") pod \"multus-gjmqz\" (UID: \"d16a97b0-104d-435b-95b9-24f19a6dbffb\") " pod="openshift-multus/multus-gjmqz" Jul 10 18:58:37.585238 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.498436 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7f479cec-2fb8-45c4-ae7d-073194a5dede-host-cni-bin\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.586179 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.498450 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7f479cec-2fb8-45c4-ae7d-073194a5dede-ovnkube-script-lib\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.586179 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.498473 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/77ed8869-fdfb-453b-b2ca-c8268fd0d0aa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l298r\" (UID: \"77ed8869-fdfb-453b-b2ca-c8268fd0d0aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l298r" Jul 10 18:58:37.586179 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.498502 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c807635a-eda8-4c7d-84bb-2f9da8c355ea-etc-sysctl-conf\") pod \"tuned-k6l5w\" (UID: \"c807635a-eda8-4c7d-84bb-2f9da8c355ea\") " pod="openshift-cluster-node-tuning-operator/tuned-k6l5w" Jul 10 18:58:37.586179 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.498521 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4grnt\" (UniqueName: \"kubernetes.io/projected/c807635a-eda8-4c7d-84bb-2f9da8c355ea-kube-api-access-4grnt\") pod \"tuned-k6l5w\" (UID: \"c807635a-eda8-4c7d-84bb-2f9da8c355ea\") " pod="openshift-cluster-node-tuning-operator/tuned-k6l5w" Jul 10 18:58:37.586179 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.498539 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7f479cec-2fb8-45c4-ae7d-073194a5dede-host-cni-netd\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.586179 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.498564 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71f50128-1d81-4578-8758-cc4b92443331-metrics-certs\") pod \"network-metrics-daemon-brfp2\" (UID: \"71f50128-1d81-4578-8758-cc4b92443331\") " pod="openshift-multus/network-metrics-daemon-brfp2" Jul 10 18:58:37.586179 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.498591 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe-original-pull-secret\") pod \"global-pull-secret-syncer-qc4kd\" (UID: \"f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe\") " pod="kube-system/global-pull-secret-syncer-qc4kd" Jul 10 18:58:37.586179 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.498609 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/77ed8869-fdfb-453b-b2ca-c8268fd0d0aa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l298r\" (UID: \"77ed8869-fdfb-453b-b2ca-c8268fd0d0aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l298r" Jul 10 18:58:37.586179 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.501962 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2c842677-9d8d-4210-9a3b-ff32bf8bda1e-agent-certs\") pod \"konnectivity-agent-zlj4w\" (UID: \"2c842677-9d8d-4210-9a3b-ff32bf8bda1e\") " pod="kube-system/konnectivity-agent-zlj4w" Jul 10 18:58:37.586179 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.510627 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpgzt\" (UniqueName: \"kubernetes.io/projected/77ed8869-fdfb-453b-b2ca-c8268fd0d0aa-kube-api-access-zpgzt\") pod \"aws-ebs-csi-driver-node-l298r\" (UID: \"77ed8869-fdfb-453b-b2ca-c8268fd0d0aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l298r" Jul 10 18:58:37.752586 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.598843 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d16a97b0-104d-435b-95b9-24f19a6dbffb-multus-conf-dir\") pod \"multus-gjmqz\" (UID: \"d16a97b0-104d-435b-95b9-24f19a6dbffb\") " pod="openshift-multus/multus-gjmqz" Jul 10 18:58:37.752586 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.598884 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c807635a-eda8-4c7d-84bb-2f9da8c355ea-etc-tuned\") pod \"tuned-k6l5w\" (UID: \"c807635a-eda8-4c7d-84bb-2f9da8c355ea\") " pod="openshift-cluster-node-tuning-operator/tuned-k6l5w" Jul 10 18:58:37.752586 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.598944 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d16a97b0-104d-435b-95b9-24f19a6dbffb-multus-conf-dir\") pod \"multus-gjmqz\" (UID: \"d16a97b0-104d-435b-95b9-24f19a6dbffb\") " pod="openshift-multus/multus-gjmqz" Jul 10 18:58:37.752586 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.599010 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f479cec-2fb8-45c4-ae7d-073194a5dede-host-run-ovn-kubernetes\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.752586 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.599051 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f479cec-2fb8-45c4-ae7d-073194a5dede-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.752586 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.599085 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7f479cec-2fb8-45c4-ae7d-073194a5dede-ovnkube-config\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.752586 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.599111 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7f479cec-2fb8-45c4-ae7d-073194a5dede-ovn-node-metrics-cert\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.752586 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.599114 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f479cec-2fb8-45c4-ae7d-073194a5dede-host-run-ovn-kubernetes\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.752586 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.599114 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f479cec-2fb8-45c4-ae7d-073194a5dede-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.752586 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.599138 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe-kubelet-config\") pod \"global-pull-secret-syncer-qc4kd\" (UID: \"f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe\") " pod="kube-system/global-pull-secret-syncer-qc4kd" Jul 10 18:58:37.752586 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.599166 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8cb72f0a-5a49-4633-8ca4-3a4e240a4aab-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zxs79\" (UID: \"8cb72f0a-5a49-4633-8ca4-3a4e240a4aab\") " pod="openshift-multus/multus-additional-cni-plugins-zxs79" Jul 10 18:58:37.752586 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.599209 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f479cec-2fb8-45c4-ae7d-073194a5dede-etc-openvswitch\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.752586 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.599242 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxpbj\" (UniqueName: \"kubernetes.io/projected/7f479cec-2fb8-45c4-ae7d-073194a5dede-kube-api-access-fxpbj\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.752586 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.599211 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe-kubelet-config\") pod \"global-pull-secret-syncer-qc4kd\" (UID: \"f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe\") " pod="kube-system/global-pull-secret-syncer-qc4kd" Jul 10 18:58:37.752586 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.599269 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d16a97b0-104d-435b-95b9-24f19a6dbffb-system-cni-dir\") pod \"multus-gjmqz\" (UID: \"d16a97b0-104d-435b-95b9-24f19a6dbffb\") " pod="openshift-multus/multus-gjmqz" Jul 10 18:58:37.752586 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.599270 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f479cec-2fb8-45c4-ae7d-073194a5dede-etc-openvswitch\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.752586 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.599340 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d16a97b0-104d-435b-95b9-24f19a6dbffb-host-var-lib-kubelet\") pod \"multus-gjmqz\" (UID: \"d16a97b0-104d-435b-95b9-24f19a6dbffb\") " pod="openshift-multus/multus-gjmqz" Jul 10 18:58:37.754593 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.599379 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d16a97b0-104d-435b-95b9-24f19a6dbffb-system-cni-dir\") pod \"multus-gjmqz\" (UID: \"d16a97b0-104d-435b-95b9-24f19a6dbffb\") " pod="openshift-multus/multus-gjmqz" Jul 10 18:58:37.754593 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.599390 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwxth\" (UniqueName: \"kubernetes.io/projected/48523b16-8cba-40d8-bbaf-e05814fcb820-kube-api-access-vwxth\") pod \"network-check-target-4fmmt\" (UID: \"48523b16-8cba-40d8-bbaf-e05814fcb820\") " pod="openshift-network-diagnostics/network-check-target-4fmmt" Jul 10 18:58:37.754593 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.599457 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7814607d-ae88-4915-bc20-345f93f50914-host-slash\") pod \"iptables-alerter-5z5v5\" (UID: \"7814607d-ae88-4915-bc20-345f93f50914\") " pod="openshift-network-operator/iptables-alerter-5z5v5" Jul 10 18:58:37.754593 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.599486 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7f479cec-2fb8-45c4-ae7d-073194a5dede-log-socket\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.754593 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.599512 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5l2l\" (UniqueName: \"kubernetes.io/projected/10a5a151-ab9d-45db-8502-26d00ec21659-kube-api-access-f5l2l\") pod \"node-ca-7btwf\" (UID: \"10a5a151-ab9d-45db-8502-26d00ec21659\") " pod="openshift-image-registry/node-ca-7btwf" Jul 10 18:58:37.754593 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.599538 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d16a97b0-104d-435b-95b9-24f19a6dbffb-host-var-lib-cni-bin\") pod \"multus-gjmqz\" (UID: \"d16a97b0-104d-435b-95b9-24f19a6dbffb\") " pod="openshift-multus/multus-gjmqz" Jul 10 18:58:37.754593 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.599574 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c807635a-eda8-4c7d-84bb-2f9da8c355ea-etc-systemd\") pod \"tuned-k6l5w\" (UID: \"c807635a-eda8-4c7d-84bb-2f9da8c355ea\") " pod="openshift-cluster-node-tuning-operator/tuned-k6l5w" Jul 10 18:58:37.754593 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.599598 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c807635a-eda8-4c7d-84bb-2f9da8c355ea-lib-modules\") pod \"tuned-k6l5w\" (UID: \"c807635a-eda8-4c7d-84bb-2f9da8c355ea\") " pod="openshift-cluster-node-tuning-operator/tuned-k6l5w" Jul 10 18:58:37.754593 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.599609 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7f479cec-2fb8-45c4-ae7d-073194a5dede-log-socket\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.754593 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.599623 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/df38c8b2-26d6-46bb-95ca-166ff4723ceb-hosts-file\") pod \"node-resolver-zf82d\" (UID: \"df38c8b2-26d6-46bb-95ca-166ff4723ceb\") " pod="openshift-dns/node-resolver-zf82d" Jul 10 18:58:37.754593 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.599573 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d16a97b0-104d-435b-95b9-24f19a6dbffb-host-var-lib-kubelet\") pod \"multus-gjmqz\" (UID: \"d16a97b0-104d-435b-95b9-24f19a6dbffb\") " pod="openshift-multus/multus-gjmqz" Jul 10 18:58:37.754593 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.600081 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8cb72f0a-5a49-4633-8ca4-3a4e240a4aab-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zxs79\" (UID: \"8cb72f0a-5a49-4633-8ca4-3a4e240a4aab\") " pod="openshift-multus/multus-additional-cni-plugins-zxs79" Jul 10 18:58:37.754593 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.600211 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d16a97b0-104d-435b-95b9-24f19a6dbffb-host-var-lib-cni-bin\") pod \"multus-gjmqz\" (UID: \"d16a97b0-104d-435b-95b9-24f19a6dbffb\") " pod="openshift-multus/multus-gjmqz" Jul 10 18:58:37.754593 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.600217 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c807635a-eda8-4c7d-84bb-2f9da8c355ea-etc-systemd\") pod \"tuned-k6l5w\" (UID: \"c807635a-eda8-4c7d-84bb-2f9da8c355ea\") " pod="openshift-cluster-node-tuning-operator/tuned-k6l5w" Jul 10 18:58:37.754593 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.600248 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7f479cec-2fb8-45c4-ae7d-073194a5dede-host-slash\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.754593 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.600260 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7814607d-ae88-4915-bc20-345f93f50914-host-slash\") pod \"iptables-alerter-5z5v5\" (UID: \"7814607d-ae88-4915-bc20-345f93f50914\") " pod="openshift-network-operator/iptables-alerter-5z5v5" Jul 10 18:58:37.754593 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.600278 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/10a5a151-ab9d-45db-8502-26d00ec21659-host\") pod \"node-ca-7btwf\" (UID: \"10a5a151-ab9d-45db-8502-26d00ec21659\") " pod="openshift-image-registry/node-ca-7btwf" Jul 10 18:58:37.754593 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.600314 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c807635a-eda8-4c7d-84bb-2f9da8c355ea-lib-modules\") pod \"tuned-k6l5w\" (UID: \"c807635a-eda8-4c7d-84bb-2f9da8c355ea\") " pod="openshift-cluster-node-tuning-operator/tuned-k6l5w" Jul 10 18:58:37.755360 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.600394 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8cb72f0a-5a49-4633-8ca4-3a4e240a4aab-system-cni-dir\") pod \"multus-additional-cni-plugins-zxs79\" (UID: \"8cb72f0a-5a49-4633-8ca4-3a4e240a4aab\") " pod="openshift-multus/multus-additional-cni-plugins-zxs79" Jul 10 18:58:37.755360 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.600422 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7f479cec-2fb8-45c4-ae7d-073194a5dede-run-ovn\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.755360 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.600450 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7f479cec-2fb8-45c4-ae7d-073194a5dede-host-slash\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.755360 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.600458 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/10a5a151-ab9d-45db-8502-26d00ec21659-host\") pod \"node-ca-7btwf\" (UID: \"10a5a151-ab9d-45db-8502-26d00ec21659\") " pod="openshift-image-registry/node-ca-7btwf" Jul 10 18:58:37.755360 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.600523 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/10a5a151-ab9d-45db-8502-26d00ec21659-serviceca\") pod \"node-ca-7btwf\" (UID: \"10a5a151-ab9d-45db-8502-26d00ec21659\") " pod="openshift-image-registry/node-ca-7btwf" Jul 10 18:58:37.755360 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.600566 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/df38c8b2-26d6-46bb-95ca-166ff4723ceb-hosts-file\") pod \"node-resolver-zf82d\" (UID: \"df38c8b2-26d6-46bb-95ca-166ff4723ceb\") " pod="openshift-dns/node-resolver-zf82d" Jul 10 18:58:37.755360 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.600611 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8cb72f0a-5a49-4633-8ca4-3a4e240a4aab-cnibin\") pod \"multus-additional-cni-plugins-zxs79\" (UID: \"8cb72f0a-5a49-4633-8ca4-3a4e240a4aab\") " pod="openshift-multus/multus-additional-cni-plugins-zxs79" Jul 10 18:58:37.755360 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.600637 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8cb72f0a-5a49-4633-8ca4-3a4e240a4aab-os-release\") pod \"multus-additional-cni-plugins-zxs79\" (UID: \"8cb72f0a-5a49-4633-8ca4-3a4e240a4aab\") " pod="openshift-multus/multus-additional-cni-plugins-zxs79" Jul 10 18:58:37.755360 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.600648 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8cb72f0a-5a49-4633-8ca4-3a4e240a4aab-cnibin\") pod \"multus-additional-cni-plugins-zxs79\" (UID: \"8cb72f0a-5a49-4633-8ca4-3a4e240a4aab\") " pod="openshift-multus/multus-additional-cni-plugins-zxs79" Jul 10 18:58:37.755360 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.600669 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d16a97b0-104d-435b-95b9-24f19a6dbffb-cni-binary-copy\") pod \"multus-gjmqz\" (UID: \"d16a97b0-104d-435b-95b9-24f19a6dbffb\") " pod="openshift-multus/multus-gjmqz" Jul 10 18:58:37.755360 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.600737 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7f479cec-2fb8-45c4-ae7d-073194a5dede-run-ovn\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.755360 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.600766 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7f479cec-2fb8-45c4-ae7d-073194a5dede-run-systemd\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.755360 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.600797 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f479cec-2fb8-45c4-ae7d-073194a5dede-run-openvswitch\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.755360 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.600842 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8cb72f0a-5a49-4633-8ca4-3a4e240a4aab-system-cni-dir\") pod \"multus-additional-cni-plugins-zxs79\" (UID: \"8cb72f0a-5a49-4633-8ca4-3a4e240a4aab\") " pod="openshift-multus/multus-additional-cni-plugins-zxs79" Jul 10 18:58:37.755360 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.600922 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7f479cec-2fb8-45c4-ae7d-073194a5dede-run-systemd\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.755360 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.600986 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8cb72f0a-5a49-4633-8ca4-3a4e240a4aab-os-release\") pod \"multus-additional-cni-plugins-zxs79\" (UID: \"8cb72f0a-5a49-4633-8ca4-3a4e240a4aab\") " pod="openshift-multus/multus-additional-cni-plugins-zxs79" Jul 10 18:58:37.755360 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.601001 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7f479cec-2fb8-45c4-ae7d-073194a5dede-ovnkube-config\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.755360 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.601040 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f479cec-2fb8-45c4-ae7d-073194a5dede-run-openvswitch\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.756077 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.601079 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe-dbus\") pod \"global-pull-secret-syncer-qc4kd\" (UID: \"f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe\") " pod="kube-system/global-pull-secret-syncer-qc4kd" Jul 10 18:58:37.756077 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.601114 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d16a97b0-104d-435b-95b9-24f19a6dbffb-multus-socket-dir-parent\") pod \"multus-gjmqz\" (UID: \"d16a97b0-104d-435b-95b9-24f19a6dbffb\") " pod="openshift-multus/multus-gjmqz" Jul 10 18:58:37.756077 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.601138 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7f479cec-2fb8-45c4-ae7d-073194a5dede-node-log\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.756077 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.601173 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7f479cec-2fb8-45c4-ae7d-073194a5dede-env-overrides\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.756077 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.601201 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8cb72f0a-5a49-4633-8ca4-3a4e240a4aab-cni-binary-copy\") pod \"multus-additional-cni-plugins-zxs79\" (UID: \"8cb72f0a-5a49-4633-8ca4-3a4e240a4aab\") " pod="openshift-multus/multus-additional-cni-plugins-zxs79" Jul 10 18:58:37.756077 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.601225 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d16a97b0-104d-435b-95b9-24f19a6dbffb-hostroot\") pod \"multus-gjmqz\" (UID: \"d16a97b0-104d-435b-95b9-24f19a6dbffb\") " pod="openshift-multus/multus-gjmqz" Jul 10 18:58:37.756077 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.601283 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d16a97b0-104d-435b-95b9-24f19a6dbffb-etc-kubernetes\") pod \"multus-gjmqz\" (UID: \"d16a97b0-104d-435b-95b9-24f19a6dbffb\") " pod="openshift-multus/multus-gjmqz" Jul 10 18:58:37.756077 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.601308 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c807635a-eda8-4c7d-84bb-2f9da8c355ea-run\") pod \"tuned-k6l5w\" (UID: \"c807635a-eda8-4c7d-84bb-2f9da8c355ea\") " pod="openshift-cluster-node-tuning-operator/tuned-k6l5w" Jul 10 18:58:37.756077 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.601331 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c807635a-eda8-4c7d-84bb-2f9da8c355ea-tmp\") pod \"tuned-k6l5w\" (UID: \"c807635a-eda8-4c7d-84bb-2f9da8c355ea\") " pod="openshift-cluster-node-tuning-operator/tuned-k6l5w" Jul 10 18:58:37.756077 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.601357 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7t8r\" (UniqueName: \"kubernetes.io/projected/71f50128-1d81-4578-8758-cc4b92443331-kube-api-access-g7t8r\") pod \"network-metrics-daemon-brfp2\" (UID: \"71f50128-1d81-4578-8758-cc4b92443331\") " pod="openshift-multus/network-metrics-daemon-brfp2" Jul 10 18:58:37.756077 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.601386 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c807635a-eda8-4c7d-84bb-2f9da8c355ea-etc-sysctl-d\") pod \"tuned-k6l5w\" (UID: \"c807635a-eda8-4c7d-84bb-2f9da8c355ea\") " pod="openshift-cluster-node-tuning-operator/tuned-k6l5w" Jul 10 18:58:37.756077 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.601391 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/10a5a151-ab9d-45db-8502-26d00ec21659-serviceca\") pod \"node-ca-7btwf\" (UID: \"10a5a151-ab9d-45db-8502-26d00ec21659\") " pod="openshift-image-registry/node-ca-7btwf" Jul 10 18:58:37.756077 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.601408 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe-dbus\") pod \"global-pull-secret-syncer-qc4kd\" (UID: \"f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe\") " pod="kube-system/global-pull-secret-syncer-qc4kd" Jul 10 18:58:37.756077 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.601282 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d16a97b0-104d-435b-95b9-24f19a6dbffb-multus-socket-dir-parent\") pod \"multus-gjmqz\" (UID: \"d16a97b0-104d-435b-95b9-24f19a6dbffb\") " pod="openshift-multus/multus-gjmqz" Jul 10 18:58:37.756077 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.601410 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d16a97b0-104d-435b-95b9-24f19a6dbffb-cni-binary-copy\") pod \"multus-gjmqz\" (UID: \"d16a97b0-104d-435b-95b9-24f19a6dbffb\") " pod="openshift-multus/multus-gjmqz" Jul 10 18:58:37.756077 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.601449 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7f479cec-2fb8-45c4-ae7d-073194a5dede-systemd-units\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.756077 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.601528 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f479cec-2fb8-45c4-ae7d-073194a5dede-var-lib-openvswitch\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.756077 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.601557 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c807635a-eda8-4c7d-84bb-2f9da8c355ea-etc-sysctl-d\") pod \"tuned-k6l5w\" (UID: \"c807635a-eda8-4c7d-84bb-2f9da8c355ea\") " pod="openshift-cluster-node-tuning-operator/tuned-k6l5w" Jul 10 18:58:37.756820 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.601566 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c807635a-eda8-4c7d-84bb-2f9da8c355ea-etc-tuned\") pod \"tuned-k6l5w\" (UID: \"c807635a-eda8-4c7d-84bb-2f9da8c355ea\") " pod="openshift-cluster-node-tuning-operator/tuned-k6l5w" Jul 10 18:58:37.756820 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.601561 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8cb72f0a-5a49-4633-8ca4-3a4e240a4aab-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zxs79\" (UID: \"8cb72f0a-5a49-4633-8ca4-3a4e240a4aab\") " pod="openshift-multus/multus-additional-cni-plugins-zxs79" Jul 10 18:58:37.756820 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.601606 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d16a97b0-104d-435b-95b9-24f19a6dbffb-host-run-multus-certs\") pod \"multus-gjmqz\" (UID: \"d16a97b0-104d-435b-95b9-24f19a6dbffb\") " pod="openshift-multus/multus-gjmqz" Jul 10 18:58:37.756820 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.601654 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk2tf\" (UniqueName: \"kubernetes.io/projected/d16a97b0-104d-435b-95b9-24f19a6dbffb-kube-api-access-vk2tf\") pod \"multus-gjmqz\" (UID: \"d16a97b0-104d-435b-95b9-24f19a6dbffb\") " pod="openshift-multus/multus-gjmqz" Jul 10 18:58:37.756820 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.601664 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7f479cec-2fb8-45c4-ae7d-073194a5dede-systemd-units\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.756820 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.601683 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c807635a-eda8-4c7d-84bb-2f9da8c355ea-var-lib-kubelet\") pod \"tuned-k6l5w\" (UID: \"c807635a-eda8-4c7d-84bb-2f9da8c355ea\") " pod="openshift-cluster-node-tuning-operator/tuned-k6l5w" Jul 10 18:58:37.756820 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.601708 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d16a97b0-104d-435b-95b9-24f19a6dbffb-etc-kubernetes\") pod \"multus-gjmqz\" (UID: \"d16a97b0-104d-435b-95b9-24f19a6dbffb\") " pod="openshift-multus/multus-gjmqz" Jul 10 18:58:37.756820 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.601722 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df38c8b2-26d6-46bb-95ca-166ff4723ceb-tmp-dir\") pod \"node-resolver-zf82d\" (UID: \"df38c8b2-26d6-46bb-95ca-166ff4723ceb\") " pod="openshift-dns/node-resolver-zf82d" Jul 10 18:58:37.756820 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.601710 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8cb72f0a-5a49-4633-8ca4-3a4e240a4aab-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zxs79\" (UID: \"8cb72f0a-5a49-4633-8ca4-3a4e240a4aab\") " pod="openshift-multus/multus-additional-cni-plugins-zxs79" Jul 10 18:58:37.756820 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.601753 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c807635a-eda8-4c7d-84bb-2f9da8c355ea-run\") pod \"tuned-k6l5w\" (UID: \"c807635a-eda8-4c7d-84bb-2f9da8c355ea\") " pod="openshift-cluster-node-tuning-operator/tuned-k6l5w" Jul 10 18:58:37.756820 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.601755 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f479cec-2fb8-45c4-ae7d-073194a5dede-host-kubelet\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.756820 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.601836 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f479cec-2fb8-45c4-ae7d-073194a5dede-var-lib-openvswitch\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.756820 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.601852 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c807635a-eda8-4c7d-84bb-2f9da8c355ea-var-lib-kubelet\") pod \"tuned-k6l5w\" (UID: \"c807635a-eda8-4c7d-84bb-2f9da8c355ea\") " pod="openshift-cluster-node-tuning-operator/tuned-k6l5w" Jul 10 18:58:37.756820 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.601869 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d16a97b0-104d-435b-95b9-24f19a6dbffb-cnibin\") pod \"multus-gjmqz\" (UID: \"d16a97b0-104d-435b-95b9-24f19a6dbffb\") " pod="openshift-multus/multus-gjmqz" Jul 10 18:58:37.756820 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.601898 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7f479cec-2fb8-45c4-ae7d-073194a5dede-node-log\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.756820 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.601906 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7f479cec-2fb8-45c4-ae7d-073194a5dede-host-cni-bin\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.756820 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.601934 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7f479cec-2fb8-45c4-ae7d-073194a5dede-ovnkube-script-lib\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.756820 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.601935 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f479cec-2fb8-45c4-ae7d-073194a5dede-host-kubelet\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.757957 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.601967 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d16a97b0-104d-435b-95b9-24f19a6dbffb-host-run-multus-certs\") pod \"multus-gjmqz\" (UID: \"d16a97b0-104d-435b-95b9-24f19a6dbffb\") " pod="openshift-multus/multus-gjmqz" Jul 10 18:58:37.757957 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.602037 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c807635a-eda8-4c7d-84bb-2f9da8c355ea-etc-sysctl-conf\") pod \"tuned-k6l5w\" (UID: \"c807635a-eda8-4c7d-84bb-2f9da8c355ea\") " pod="openshift-cluster-node-tuning-operator/tuned-k6l5w" Jul 10 18:58:37.757957 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.602067 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df38c8b2-26d6-46bb-95ca-166ff4723ceb-tmp-dir\") pod \"node-resolver-zf82d\" (UID: \"df38c8b2-26d6-46bb-95ca-166ff4723ceb\") " pod="openshift-dns/node-resolver-zf82d" Jul 10 18:58:37.757957 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.602067 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4grnt\" (UniqueName: \"kubernetes.io/projected/c807635a-eda8-4c7d-84bb-2f9da8c355ea-kube-api-access-4grnt\") pod \"tuned-k6l5w\" (UID: \"c807635a-eda8-4c7d-84bb-2f9da8c355ea\") " pod="openshift-cluster-node-tuning-operator/tuned-k6l5w" Jul 10 18:58:37.757957 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.602106 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d16a97b0-104d-435b-95b9-24f19a6dbffb-hostroot\") pod \"multus-gjmqz\" (UID: \"d16a97b0-104d-435b-95b9-24f19a6dbffb\") " pod="openshift-multus/multus-gjmqz" Jul 10 18:58:37.757957 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.602128 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7f479cec-2fb8-45c4-ae7d-073194a5dede-host-cni-netd\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.757957 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.602187 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8cb72f0a-5a49-4633-8ca4-3a4e240a4aab-cni-binary-copy\") pod \"multus-additional-cni-plugins-zxs79\" (UID: \"8cb72f0a-5a49-4633-8ca4-3a4e240a4aab\") " pod="openshift-multus/multus-additional-cni-plugins-zxs79" Jul 10 18:58:37.757957 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.602275 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71f50128-1d81-4578-8758-cc4b92443331-metrics-certs\") pod \"network-metrics-daemon-brfp2\" (UID: \"71f50128-1d81-4578-8758-cc4b92443331\") " pod="openshift-multus/network-metrics-daemon-brfp2" Jul 10 18:58:37.757957 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.602290 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d16a97b0-104d-435b-95b9-24f19a6dbffb-cnibin\") pod \"multus-gjmqz\" (UID: \"d16a97b0-104d-435b-95b9-24f19a6dbffb\") " pod="openshift-multus/multus-gjmqz" Jul 10 18:58:37.757957 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.602320 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7f479cec-2fb8-45c4-ae7d-073194a5dede-host-cni-netd\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.757957 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.602338 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe-original-pull-secret\") pod \"global-pull-secret-syncer-qc4kd\" (UID: \"f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe\") " pod="kube-system/global-pull-secret-syncer-qc4kd" Jul 10 18:58:37.757957 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.602369 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7f479cec-2fb8-45c4-ae7d-073194a5dede-host-cni-bin\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.757957 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.602499 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c807635a-eda8-4c7d-84bb-2f9da8c355ea-etc-sysctl-conf\") pod \"tuned-k6l5w\" (UID: \"c807635a-eda8-4c7d-84bb-2f9da8c355ea\") " pod="openshift-cluster-node-tuning-operator/tuned-k6l5w" Jul 10 18:58:37.757957 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:37.602505 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:58:37.757957 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.602538 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d16a97b0-104d-435b-95b9-24f19a6dbffb-host-run-k8s-cni-cncf-io\") pod \"multus-gjmqz\" (UID: \"d16a97b0-104d-435b-95b9-24f19a6dbffb\") " pod="openshift-multus/multus-gjmqz" Jul 10 18:58:37.757957 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:37.602548 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 18:58:37.757957 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:37.603039 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71f50128-1d81-4578-8758-cc4b92443331-metrics-certs podName:71f50128-1d81-4578-8758-cc4b92443331 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:38.103009052 +0000 UTC m=+2.147151556 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71f50128-1d81-4578-8758-cc4b92443331-metrics-certs") pod "network-metrics-daemon-brfp2" (UID: "71f50128-1d81-4578-8758-cc4b92443331") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:58:37.758553 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:37.603779 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe-original-pull-secret podName:f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe nodeName:}" failed. No retries permitted until 2026-07-10 18:58:38.103741266 +0000 UTC m=+2.147883756 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe-original-pull-secret") pod "global-pull-secret-syncer-qc4kd" (UID: "f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe") : object "kube-system"/"original-pull-secret" not registered Jul 10 18:58:37.758553 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.603899 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c807635a-eda8-4c7d-84bb-2f9da8c355ea-tmp\") pod \"tuned-k6l5w\" (UID: \"c807635a-eda8-4c7d-84bb-2f9da8c355ea\") " pod="openshift-cluster-node-tuning-operator/tuned-k6l5w" Jul 10 18:58:37.758553 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.603940 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d16a97b0-104d-435b-95b9-24f19a6dbffb-host-var-lib-cni-multus\") pod \"multus-gjmqz\" (UID: \"d16a97b0-104d-435b-95b9-24f19a6dbffb\") " pod="openshift-multus/multus-gjmqz" Jul 10 18:58:37.758553 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.603968 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7f479cec-2fb8-45c4-ae7d-073194a5dede-env-overrides\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.758553 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.603990 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8cb72f0a-5a49-4633-8ca4-3a4e240a4aab-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zxs79\" (UID: \"8cb72f0a-5a49-4633-8ca4-3a4e240a4aab\") " pod="openshift-multus/multus-additional-cni-plugins-zxs79" Jul 10 18:58:37.758553 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.604032 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvsjx\" (UniqueName: \"kubernetes.io/projected/8cb72f0a-5a49-4633-8ca4-3a4e240a4aab-kube-api-access-pvsjx\") pod \"multus-additional-cni-plugins-zxs79\" (UID: \"8cb72f0a-5a49-4633-8ca4-3a4e240a4aab\") " pod="openshift-multus/multus-additional-cni-plugins-zxs79" Jul 10 18:58:37.758553 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.604072 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d16a97b0-104d-435b-95b9-24f19a6dbffb-multus-cni-dir\") pod \"multus-gjmqz\" (UID: \"d16a97b0-104d-435b-95b9-24f19a6dbffb\") " pod="openshift-multus/multus-gjmqz" Jul 10 18:58:37.758553 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.604105 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d16a97b0-104d-435b-95b9-24f19a6dbffb-os-release\") pod \"multus-gjmqz\" (UID: \"d16a97b0-104d-435b-95b9-24f19a6dbffb\") " pod="openshift-multus/multus-gjmqz" Jul 10 18:58:37.758553 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.604134 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d16a97b0-104d-435b-95b9-24f19a6dbffb-multus-daemon-config\") pod \"multus-gjmqz\" (UID: \"d16a97b0-104d-435b-95b9-24f19a6dbffb\") " pod="openshift-multus/multus-gjmqz" Jul 10 18:58:37.758553 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.604169 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c807635a-eda8-4c7d-84bb-2f9da8c355ea-etc-sysconfig\") pod \"tuned-k6l5w\" (UID: \"c807635a-eda8-4c7d-84bb-2f9da8c355ea\") " pod="openshift-cluster-node-tuning-operator/tuned-k6l5w" Jul 10 18:58:37.758553 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.604203 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c807635a-eda8-4c7d-84bb-2f9da8c355ea-etc-kubernetes\") pod \"tuned-k6l5w\" (UID: \"c807635a-eda8-4c7d-84bb-2f9da8c355ea\") " pod="openshift-cluster-node-tuning-operator/tuned-k6l5w" Jul 10 18:58:37.758553 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.604210 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7f479cec-2fb8-45c4-ae7d-073194a5dede-ovnkube-script-lib\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.758553 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.604234 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c807635a-eda8-4c7d-84bb-2f9da8c355ea-sys\") pod \"tuned-k6l5w\" (UID: \"c807635a-eda8-4c7d-84bb-2f9da8c355ea\") " pod="openshift-cluster-node-tuning-operator/tuned-k6l5w" Jul 10 18:58:37.758553 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.604267 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c807635a-eda8-4c7d-84bb-2f9da8c355ea-host\") pod \"tuned-k6l5w\" (UID: \"c807635a-eda8-4c7d-84bb-2f9da8c355ea\") " pod="openshift-cluster-node-tuning-operator/tuned-k6l5w" Jul 10 18:58:37.758553 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.604292 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d16a97b0-104d-435b-95b9-24f19a6dbffb-host-run-netns\") pod \"multus-gjmqz\" (UID: \"d16a97b0-104d-435b-95b9-24f19a6dbffb\") " pod="openshift-multus/multus-gjmqz" Jul 10 18:58:37.758553 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.604329 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c807635a-eda8-4c7d-84bb-2f9da8c355ea-etc-modprobe-d\") pod \"tuned-k6l5w\" (UID: \"c807635a-eda8-4c7d-84bb-2f9da8c355ea\") " pod="openshift-cluster-node-tuning-operator/tuned-k6l5w" Jul 10 18:58:37.758553 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.604340 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7f479cec-2fb8-45c4-ae7d-073194a5dede-ovn-node-metrics-cert\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.759155 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.604366 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzkpw\" (UniqueName: \"kubernetes.io/projected/df38c8b2-26d6-46bb-95ca-166ff4723ceb-kube-api-access-xzkpw\") pod \"node-resolver-zf82d\" (UID: \"df38c8b2-26d6-46bb-95ca-166ff4723ceb\") " pod="openshift-dns/node-resolver-zf82d" Jul 10 18:58:37.759155 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.604402 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7814607d-ae88-4915-bc20-345f93f50914-iptables-alerter-script\") pod \"iptables-alerter-5z5v5\" (UID: \"7814607d-ae88-4915-bc20-345f93f50914\") " pod="openshift-network-operator/iptables-alerter-5z5v5" Jul 10 18:58:37.759155 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.604431 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsq2d\" (UniqueName: \"kubernetes.io/projected/7814607d-ae88-4915-bc20-345f93f50914-kube-api-access-vsq2d\") pod \"iptables-alerter-5z5v5\" (UID: \"7814607d-ae88-4915-bc20-345f93f50914\") " pod="openshift-network-operator/iptables-alerter-5z5v5" Jul 10 18:58:37.759155 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.604467 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7f479cec-2fb8-45c4-ae7d-073194a5dede-host-run-netns\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.759155 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.604662 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7f479cec-2fb8-45c4-ae7d-073194a5dede-host-run-netns\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.759155 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.604717 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d16a97b0-104d-435b-95b9-24f19a6dbffb-host-run-k8s-cni-cncf-io\") pod \"multus-gjmqz\" (UID: \"d16a97b0-104d-435b-95b9-24f19a6dbffb\") " pod="openshift-multus/multus-gjmqz" Jul 10 18:58:37.759155 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.604797 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d16a97b0-104d-435b-95b9-24f19a6dbffb-host-var-lib-cni-multus\") pod \"multus-gjmqz\" (UID: \"d16a97b0-104d-435b-95b9-24f19a6dbffb\") " pod="openshift-multus/multus-gjmqz" Jul 10 18:58:37.759155 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.604975 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c807635a-eda8-4c7d-84bb-2f9da8c355ea-sys\") pod \"tuned-k6l5w\" (UID: \"c807635a-eda8-4c7d-84bb-2f9da8c355ea\") " pod="openshift-cluster-node-tuning-operator/tuned-k6l5w" Jul 10 18:58:37.759155 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.605387 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d16a97b0-104d-435b-95b9-24f19a6dbffb-multus-cni-dir\") pod \"multus-gjmqz\" (UID: \"d16a97b0-104d-435b-95b9-24f19a6dbffb\") " pod="openshift-multus/multus-gjmqz" Jul 10 18:58:37.759155 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.605455 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c807635a-eda8-4c7d-84bb-2f9da8c355ea-etc-modprobe-d\") pod \"tuned-k6l5w\" (UID: \"c807635a-eda8-4c7d-84bb-2f9da8c355ea\") " pod="openshift-cluster-node-tuning-operator/tuned-k6l5w" Jul 10 18:58:37.759155 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.605475 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d16a97b0-104d-435b-95b9-24f19a6dbffb-os-release\") pod \"multus-gjmqz\" (UID: \"d16a97b0-104d-435b-95b9-24f19a6dbffb\") " pod="openshift-multus/multus-gjmqz" Jul 10 18:58:37.759155 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.605533 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d16a97b0-104d-435b-95b9-24f19a6dbffb-host-run-netns\") pod \"multus-gjmqz\" (UID: \"d16a97b0-104d-435b-95b9-24f19a6dbffb\") " pod="openshift-multus/multus-gjmqz" Jul 10 18:58:37.759155 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.605545 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c807635a-eda8-4c7d-84bb-2f9da8c355ea-host\") pod \"tuned-k6l5w\" (UID: \"c807635a-eda8-4c7d-84bb-2f9da8c355ea\") " pod="openshift-cluster-node-tuning-operator/tuned-k6l5w" Jul 10 18:58:37.759155 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.605946 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8cb72f0a-5a49-4633-8ca4-3a4e240a4aab-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zxs79\" (UID: \"8cb72f0a-5a49-4633-8ca4-3a4e240a4aab\") " pod="openshift-multus/multus-additional-cni-plugins-zxs79" Jul 10 18:58:37.759155 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.606095 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d16a97b0-104d-435b-95b9-24f19a6dbffb-multus-daemon-config\") pod \"multus-gjmqz\" (UID: \"d16a97b0-104d-435b-95b9-24f19a6dbffb\") " pod="openshift-multus/multus-gjmqz" Jul 10 18:58:37.759155 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.606167 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c807635a-eda8-4c7d-84bb-2f9da8c355ea-etc-sysconfig\") pod \"tuned-k6l5w\" (UID: \"c807635a-eda8-4c7d-84bb-2f9da8c355ea\") " pod="openshift-cluster-node-tuning-operator/tuned-k6l5w" Jul 10 18:58:37.759155 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.606215 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c807635a-eda8-4c7d-84bb-2f9da8c355ea-etc-kubernetes\") pod \"tuned-k6l5w\" (UID: \"c807635a-eda8-4c7d-84bb-2f9da8c355ea\") " pod="openshift-cluster-node-tuning-operator/tuned-k6l5w" Jul 10 18:58:37.759155 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.606945 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7814607d-ae88-4915-bc20-345f93f50914-iptables-alerter-script\") pod \"iptables-alerter-5z5v5\" (UID: \"7814607d-ae88-4915-bc20-345f93f50914\") " pod="openshift-network-operator/iptables-alerter-5z5v5" Jul 10 18:58:37.759854 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:37.609002 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 18:58:37.759854 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:37.609019 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 18:58:37.759854 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:37.609038 2581 projected.go:196] Error preparing data for projected volume kube-api-access-vwxth for pod openshift-network-diagnostics/network-check-target-4fmmt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:58:37.759854 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:37.609133 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48523b16-8cba-40d8-bbaf-e05814fcb820-kube-api-access-vwxth podName:48523b16-8cba-40d8-bbaf-e05814fcb820 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:38.109113789 +0000 UTC m=+2.153256288 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vwxth" (UniqueName: "kubernetes.io/projected/48523b16-8cba-40d8-bbaf-e05814fcb820-kube-api-access-vwxth") pod "network-check-target-4fmmt" (UID: "48523b16-8cba-40d8-bbaf-e05814fcb820") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:58:37.759854 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:37.609158 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2729654f7488332f5966486449c5669.slice/crio-f8a6f4dcc8c0498c57aa993341aa69460e978239f3ce5a1bd54673051cc6ff8f WatchSource:0}: Error finding container f8a6f4dcc8c0498c57aa993341aa69460e978239f3ce5a1bd54673051cc6ff8f: Status 404 returned error can't find the container with id f8a6f4dcc8c0498c57aa993341aa69460e978239f3ce5a1bd54673051cc6ff8f Jul 10 18:58:37.759854 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.612481 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 18:58:37.759854 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.612487 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxpbj\" (UniqueName: \"kubernetes.io/projected/7f479cec-2fb8-45c4-ae7d-073194a5dede-kube-api-access-fxpbj\") pod \"ovnkube-node-69hnr\" (UID: \"7f479cec-2fb8-45c4-ae7d-073194a5dede\") " pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:37.759854 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.612697 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5l2l\" (UniqueName: \"kubernetes.io/projected/10a5a151-ab9d-45db-8502-26d00ec21659-kube-api-access-f5l2l\") pod \"node-ca-7btwf\" (UID: \"10a5a151-ab9d-45db-8502-26d00ec21659\") " pod="openshift-image-registry/node-ca-7btwf" Jul 10 18:58:37.759854 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.613799 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7btwf" Jul 10 18:58:37.759854 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.615438 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7t8r\" (UniqueName: \"kubernetes.io/projected/71f50128-1d81-4578-8758-cc4b92443331-kube-api-access-g7t8r\") pod \"network-metrics-daemon-brfp2\" (UID: \"71f50128-1d81-4578-8758-cc4b92443331\") " pod="openshift-multus/network-metrics-daemon-brfp2" Jul 10 18:58:37.759854 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.620753 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk2tf\" (UniqueName: \"kubernetes.io/projected/d16a97b0-104d-435b-95b9-24f19a6dbffb-kube-api-access-vk2tf\") pod \"multus-gjmqz\" (UID: \"d16a97b0-104d-435b-95b9-24f19a6dbffb\") " pod="openshift-multus/multus-gjmqz" Jul 10 18:58:37.759854 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.620778 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvsjx\" (UniqueName: \"kubernetes.io/projected/8cb72f0a-5a49-4633-8ca4-3a4e240a4aab-kube-api-access-pvsjx\") pod \"multus-additional-cni-plugins-zxs79\" (UID: \"8cb72f0a-5a49-4633-8ca4-3a4e240a4aab\") " pod="openshift-multus/multus-additional-cni-plugins-zxs79" Jul 10 18:58:37.759854 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.620817 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4grnt\" (UniqueName: \"kubernetes.io/projected/c807635a-eda8-4c7d-84bb-2f9da8c355ea-kube-api-access-4grnt\") pod \"tuned-k6l5w\" (UID: \"c807635a-eda8-4c7d-84bb-2f9da8c355ea\") " pod="openshift-cluster-node-tuning-operator/tuned-k6l5w" Jul 10 18:58:37.759854 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.620868 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsq2d\" (UniqueName: \"kubernetes.io/projected/7814607d-ae88-4915-bc20-345f93f50914-kube-api-access-vsq2d\") pod \"iptables-alerter-5z5v5\" (UID: \"7814607d-ae88-4915-bc20-345f93f50914\") " pod="openshift-network-operator/iptables-alerter-5z5v5" Jul 10 18:58:37.759854 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.621286 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzkpw\" (UniqueName: \"kubernetes.io/projected/df38c8b2-26d6-46bb-95ca-166ff4723ceb-kube-api-access-xzkpw\") pod \"node-resolver-zf82d\" (UID: \"df38c8b2-26d6-46bb-95ca-166ff4723ceb\") " pod="openshift-dns/node-resolver-zf82d" Jul 10 18:58:37.759854 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.622943 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zxs79" Jul 10 18:58:37.759854 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.629584 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gjmqz" Jul 10 18:58:37.759854 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:37.631488 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cb72f0a_5a49_4633_8ca4_3a4e240a4aab.slice/crio-d9a45ac2bc95065e72dc836da0ac9876a013c6cb0f1256112943f0281d8c2c57 WatchSource:0}: Error finding container d9a45ac2bc95065e72dc836da0ac9876a013c6cb0f1256112943f0281d8c2c57: Status 404 returned error can't find the container with id d9a45ac2bc95065e72dc836da0ac9876a013c6cb0f1256112943f0281d8c2c57 Jul 10 18:58:37.761140 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:37.638253 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd16a97b0_104d_435b_95b9_24f19a6dbffb.slice/crio-3623a98fe8ac2cafaf587d1fcd4c01c5382ec4b5ce6d9e19f136b50cdd411d43 WatchSource:0}: Error finding container 3623a98fe8ac2cafaf587d1fcd4c01c5382ec4b5ce6d9e19f136b50cdd411d43: Status 404 returned error can't find the container with id 3623a98fe8ac2cafaf587d1fcd4c01c5382ec4b5ce6d9e19f136b50cdd411d43 Jul 10 18:58:37.761140 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.713235 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zlj4w" Jul 10 18:58:37.761140 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:37.722379 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c842677_9d8d_4210_9a3b_ff32bf8bda1e.slice/crio-4138559c7e382aa25fcca3aa4f0b4ffa0bd82157098c151be744429c88ecc601 WatchSource:0}: Error finding container 4138559c7e382aa25fcca3aa4f0b4ffa0bd82157098c151be744429c88ecc601: Status 404 returned error can't find the container with id 4138559c7e382aa25fcca3aa4f0b4ffa0bd82157098c151be744429c88ecc601 Jul 10 18:58:37.908369 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.765452 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l298r" Jul 10 18:58:37.908369 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:37.774383 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77ed8869_fdfb_453b_b2ca_c8268fd0d0aa.slice/crio-e0f9d04639e02e5852cea3cd43769f8c991fe17ec1f1b465de3c75f8b0bc793e WatchSource:0}: Error finding container e0f9d04639e02e5852cea3cd43769f8c991fe17ec1f1b465de3c75f8b0bc793e: Status 404 returned error can't find the container with id e0f9d04639e02e5852cea3cd43769f8c991fe17ec1f1b465de3c75f8b0bc793e Jul 10 18:58:37.908369 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.814214 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-k6l5w" Jul 10 18:58:37.908369 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.821750 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zf82d" Jul 10 18:58:37.908369 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:37.822014 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc807635a_eda8_4c7d_84bb_2f9da8c355ea.slice/crio-cbb417bd28e033001b56ad0ddb360b95cc416df49367afccc40dcf1ba5e4d7c6 WatchSource:0}: Error finding container cbb417bd28e033001b56ad0ddb360b95cc416df49367afccc40dcf1ba5e4d7c6: Status 404 returned error can't find the container with id cbb417bd28e033001b56ad0ddb360b95cc416df49367afccc40dcf1ba5e4d7c6 Jul 10 18:58:37.908369 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:37.833980 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf38c8b2_26d6_46bb_95ca_166ff4723ceb.slice/crio-ed5d28504bc4585232ac77fb439cd64985945501910a5510028464d906bae799 WatchSource:0}: Error finding container ed5d28504bc4585232ac77fb439cd64985945501910a5510028464d906bae799: Status 404 returned error can't find the container with id ed5d28504bc4585232ac77fb439cd64985945501910a5510028464d906bae799 Jul 10 18:58:37.908369 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.899925 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5z5v5" Jul 10 18:58:37.909805 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:58:37.908780 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7814607d_ae88_4915_bc20_345f93f50914.slice/crio-d4e0581eaa9f6dbc12d4673eac3986f65859e7e8364bd73388daab9053d7f81d WatchSource:0}: Error finding container d4e0581eaa9f6dbc12d4673eac3986f65859e7e8364bd73388daab9053d7f81d: Status 404 returned error can't find the container with id d4e0581eaa9f6dbc12d4673eac3986f65859e7e8364bd73388daab9053d7f81d Jul 10 18:58:37.912773 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:37.912441 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:58:38.122646 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:38.109050 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71f50128-1d81-4578-8758-cc4b92443331-metrics-certs\") pod \"network-metrics-daemon-brfp2\" (UID: \"71f50128-1d81-4578-8758-cc4b92443331\") " pod="openshift-multus/network-metrics-daemon-brfp2" Jul 10 18:58:38.122646 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:38.109096 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe-original-pull-secret\") pod \"global-pull-secret-syncer-qc4kd\" (UID: \"f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe\") " pod="kube-system/global-pull-secret-syncer-qc4kd" Jul 10 18:58:38.122646 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:38.109141 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwxth\" (UniqueName: \"kubernetes.io/projected/48523b16-8cba-40d8-bbaf-e05814fcb820-kube-api-access-vwxth\") pod \"network-check-target-4fmmt\" (UID: \"48523b16-8cba-40d8-bbaf-e05814fcb820\") " pod="openshift-network-diagnostics/network-check-target-4fmmt" Jul 10 18:58:38.122646 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:38.109308 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 18:58:38.122646 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:38.109325 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 18:58:38.122646 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:38.109338 2581 projected.go:196] Error preparing data for projected volume kube-api-access-vwxth for pod openshift-network-diagnostics/network-check-target-4fmmt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:58:38.122646 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:38.109398 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48523b16-8cba-40d8-bbaf-e05814fcb820-kube-api-access-vwxth podName:48523b16-8cba-40d8-bbaf-e05814fcb820 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:39.109379146 +0000 UTC m=+3.153521630 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vwxth" (UniqueName: "kubernetes.io/projected/48523b16-8cba-40d8-bbaf-e05814fcb820-kube-api-access-vwxth") pod "network-check-target-4fmmt" (UID: "48523b16-8cba-40d8-bbaf-e05814fcb820") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:58:38.122646 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:38.109830 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:58:38.122646 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:38.109875 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71f50128-1d81-4578-8758-cc4b92443331-metrics-certs podName:71f50128-1d81-4578-8758-cc4b92443331 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:39.109861865 +0000 UTC m=+3.154004352 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71f50128-1d81-4578-8758-cc4b92443331-metrics-certs") pod "network-metrics-daemon-brfp2" (UID: "71f50128-1d81-4578-8758-cc4b92443331") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:58:38.122646 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:38.109949 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 18:58:38.122646 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:38.109980 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe-original-pull-secret podName:f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe nodeName:}" failed. No retries permitted until 2026-07-10 18:58:39.109969 +0000 UTC m=+3.154111480 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe-original-pull-secret") pod "global-pull-secret-syncer-qc4kd" (UID: "f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe") : object "kube-system"/"original-pull-secret" not registered Jul 10 18:58:38.212716 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:38.211246 2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 18:58:38.496563 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:38.442496 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 18:53:38 +0000 UTC" deadline="2028-03-16 19:59:05.093578367 +0000 UTC" Jul 10 18:58:38.496563 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:38.442535 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14761h0m26.651047491s" Jul 10 18:58:38.572075 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:38.572016 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-67.ec2.internal" event={"ID":"e2729654f7488332f5966486449c5669","Type":"ContainerStarted","Data":"f8a6f4dcc8c0498c57aa993341aa69460e978239f3ce5a1bd54673051cc6ff8f"} Jul 10 18:58:38.586167 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:38.586127 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5z5v5" event={"ID":"7814607d-ae88-4915-bc20-345f93f50914","Type":"ContainerStarted","Data":"d4e0581eaa9f6dbc12d4673eac3986f65859e7e8364bd73388daab9053d7f81d"} Jul 10 18:58:38.600607 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:38.600508 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zf82d" event={"ID":"df38c8b2-26d6-46bb-95ca-166ff4723ceb","Type":"ContainerStarted","Data":"ed5d28504bc4585232ac77fb439cd64985945501910a5510028464d906bae799"} Jul 10 18:58:38.604022 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:38.603936 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-k6l5w" event={"ID":"c807635a-eda8-4c7d-84bb-2f9da8c355ea","Type":"ContainerStarted","Data":"cbb417bd28e033001b56ad0ddb360b95cc416df49367afccc40dcf1ba5e4d7c6"} Jul 10 18:58:38.607806 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:38.607732 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zlj4w" event={"ID":"2c842677-9d8d-4210-9a3b-ff32bf8bda1e","Type":"ContainerStarted","Data":"4138559c7e382aa25fcca3aa4f0b4ffa0bd82157098c151be744429c88ecc601"} Jul 10 18:58:38.616509 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:38.616446 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-67.ec2.internal" event={"ID":"a2521e690c042ce0e26fc7818bf7a692","Type":"ContainerStarted","Data":"a5bdd05b4a56fbdaac342580ca58f184a15e3d83d977a1b38f0a482bdf5155e6"} Jul 10 18:58:38.618570 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:38.618546 2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 18:58:38.628884 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:38.628745 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" event={"ID":"7f479cec-2fb8-45c4-ae7d-073194a5dede","Type":"ContainerStarted","Data":"ca2f16c5d344030e7727c26a30f0450722120be26d8eb9ea81f94d9a154f9602"} Jul 10 18:58:38.649266 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:38.649184 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l298r" event={"ID":"77ed8869-fdfb-453b-b2ca-c8268fd0d0aa","Type":"ContainerStarted","Data":"e0f9d04639e02e5852cea3cd43769f8c991fe17ec1f1b465de3c75f8b0bc793e"} Jul 10 18:58:38.667333 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:38.667013 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gjmqz" event={"ID":"d16a97b0-104d-435b-95b9-24f19a6dbffb","Type":"ContainerStarted","Data":"3623a98fe8ac2cafaf587d1fcd4c01c5382ec4b5ce6d9e19f136b50cdd411d43"} Jul 10 18:58:38.687554 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:38.687518 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zxs79" event={"ID":"8cb72f0a-5a49-4633-8ca4-3a4e240a4aab","Type":"ContainerStarted","Data":"d9a45ac2bc95065e72dc836da0ac9876a013c6cb0f1256112943f0281d8c2c57"} Jul 10 18:58:38.693603 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:38.693568 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7btwf" event={"ID":"10a5a151-ab9d-45db-8502-26d00ec21659","Type":"ContainerStarted","Data":"2bb0fce3bc06c82eace143f20fc8afe8fed3e7bfad21bf96fb883bf01eb0a1ff"} Jul 10 18:58:39.124001 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:39.123958 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71f50128-1d81-4578-8758-cc4b92443331-metrics-certs\") pod \"network-metrics-daemon-brfp2\" (UID: \"71f50128-1d81-4578-8758-cc4b92443331\") " pod="openshift-multus/network-metrics-daemon-brfp2" Jul 10 18:58:39.124261 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:39.124020 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe-original-pull-secret\") pod \"global-pull-secret-syncer-qc4kd\" (UID: \"f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe\") " pod="kube-system/global-pull-secret-syncer-qc4kd" Jul 10 18:58:39.124261 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:39.124077 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwxth\" (UniqueName: \"kubernetes.io/projected/48523b16-8cba-40d8-bbaf-e05814fcb820-kube-api-access-vwxth\") pod \"network-check-target-4fmmt\" (UID: \"48523b16-8cba-40d8-bbaf-e05814fcb820\") " pod="openshift-network-diagnostics/network-check-target-4fmmt" Jul 10 18:58:39.124261 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:39.124236 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 18:58:39.124261 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:39.124259 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 18:58:39.124480 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:39.124272 2581 projected.go:196] Error preparing data for projected volume kube-api-access-vwxth for pod openshift-network-diagnostics/network-check-target-4fmmt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:58:39.124480 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:39.124338 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48523b16-8cba-40d8-bbaf-e05814fcb820-kube-api-access-vwxth podName:48523b16-8cba-40d8-bbaf-e05814fcb820 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:41.124314712 +0000 UTC m=+5.168457193 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vwxth" (UniqueName: "kubernetes.io/projected/48523b16-8cba-40d8-bbaf-e05814fcb820-kube-api-access-vwxth") pod "network-check-target-4fmmt" (UID: "48523b16-8cba-40d8-bbaf-e05814fcb820") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:58:39.124840 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:39.124814 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:58:39.124938 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:39.124888 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71f50128-1d81-4578-8758-cc4b92443331-metrics-certs podName:71f50128-1d81-4578-8758-cc4b92443331 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:41.124862079 +0000 UTC m=+5.169004565 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71f50128-1d81-4578-8758-cc4b92443331-metrics-certs") pod "network-metrics-daemon-brfp2" (UID: "71f50128-1d81-4578-8758-cc4b92443331") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:58:39.125005 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:39.124951 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 18:58:39.125005 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:39.124987 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe-original-pull-secret podName:f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe nodeName:}" failed. No retries permitted until 2026-07-10 18:58:41.124975935 +0000 UTC m=+5.169118415 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe-original-pull-secret") pod "global-pull-secret-syncer-qc4kd" (UID: "f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe") : object "kube-system"/"original-pull-secret" not registered Jul 10 18:58:39.443926 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:39.443774 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 18:53:38 +0000 UTC" deadline="2028-02-09 23:37:05.615245694 +0000 UTC" Jul 10 18:58:39.443926 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:39.443849 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13900h38m26.171401562s" Jul 10 18:58:39.536855 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:39.536282 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qc4kd" Jul 10 18:58:39.536855 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:39.536418 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qc4kd" podUID="f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe" Jul 10 18:58:39.536855 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:39.536523 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4fmmt" Jul 10 18:58:39.536855 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:39.536647 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4fmmt" podUID="48523b16-8cba-40d8-bbaf-e05814fcb820" Jul 10 18:58:39.536855 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:39.536774 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-brfp2" Jul 10 18:58:39.537559 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:39.536872 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-brfp2" podUID="71f50128-1d81-4578-8758-cc4b92443331" Jul 10 18:58:40.249627 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:40.249596 2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 18:58:41.141513 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:41.141427 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71f50128-1d81-4578-8758-cc4b92443331-metrics-certs\") pod \"network-metrics-daemon-brfp2\" (UID: \"71f50128-1d81-4578-8758-cc4b92443331\") " pod="openshift-multus/network-metrics-daemon-brfp2" Jul 10 18:58:41.141513 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:41.141483 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe-original-pull-secret\") pod \"global-pull-secret-syncer-qc4kd\" (UID: \"f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe\") " pod="kube-system/global-pull-secret-syncer-qc4kd" Jul 10 18:58:41.142144 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:41.141534 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwxth\" (UniqueName: \"kubernetes.io/projected/48523b16-8cba-40d8-bbaf-e05814fcb820-kube-api-access-vwxth\") pod \"network-check-target-4fmmt\" (UID: \"48523b16-8cba-40d8-bbaf-e05814fcb820\") " pod="openshift-network-diagnostics/network-check-target-4fmmt" Jul 10 18:58:41.142144 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:41.141702 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 18:58:41.142144 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:41.141737 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 18:58:41.142144 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:41.141752 2581 projected.go:196] Error preparing data for projected volume kube-api-access-vwxth for pod openshift-network-diagnostics/network-check-target-4fmmt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:58:41.142144 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:41.141814 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48523b16-8cba-40d8-bbaf-e05814fcb820-kube-api-access-vwxth podName:48523b16-8cba-40d8-bbaf-e05814fcb820 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:45.141794876 +0000 UTC m=+9.185937360 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vwxth" (UniqueName: "kubernetes.io/projected/48523b16-8cba-40d8-bbaf-e05814fcb820-kube-api-access-vwxth") pod "network-check-target-4fmmt" (UID: "48523b16-8cba-40d8-bbaf-e05814fcb820") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:58:41.142442 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:41.142235 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:58:41.142442 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:41.142283 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71f50128-1d81-4578-8758-cc4b92443331-metrics-certs podName:71f50128-1d81-4578-8758-cc4b92443331 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:45.142267283 +0000 UTC m=+9.186409767 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71f50128-1d81-4578-8758-cc4b92443331-metrics-certs") pod "network-metrics-daemon-brfp2" (UID: "71f50128-1d81-4578-8758-cc4b92443331") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:58:41.142442 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:41.142345 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 18:58:41.142442 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:41.142380 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe-original-pull-secret podName:f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe nodeName:}" failed. No retries permitted until 2026-07-10 18:58:45.142367706 +0000 UTC m=+9.186510189 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe-original-pull-secret") pod "global-pull-secret-syncer-qc4kd" (UID: "f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe") : object "kube-system"/"original-pull-secret" not registered Jul 10 18:58:41.536432 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:41.536397 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qc4kd" Jul 10 18:58:41.536611 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:41.536430 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4fmmt" Jul 10 18:58:41.536611 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:41.536539 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qc4kd" podUID="f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe" Jul 10 18:58:41.537066 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:41.537006 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4fmmt" podUID="48523b16-8cba-40d8-bbaf-e05814fcb820" Jul 10 18:58:41.537066 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:41.537065 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-brfp2" Jul 10 18:58:41.537221 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:41.537160 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-brfp2" podUID="71f50128-1d81-4578-8758-cc4b92443331" Jul 10 18:58:43.536710 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:43.536665 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qc4kd" Jul 10 18:58:43.537209 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:43.536710 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4fmmt" Jul 10 18:58:43.537209 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:43.536710 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-brfp2" Jul 10 18:58:43.537209 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:43.536842 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qc4kd" podUID="f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe" Jul 10 18:58:43.537209 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:43.536991 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4fmmt" podUID="48523b16-8cba-40d8-bbaf-e05814fcb820" Jul 10 18:58:43.537209 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:43.537171 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-brfp2" podUID="71f50128-1d81-4578-8758-cc4b92443331" Jul 10 18:58:45.181072 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:45.180972 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwxth\" (UniqueName: \"kubernetes.io/projected/48523b16-8cba-40d8-bbaf-e05814fcb820-kube-api-access-vwxth\") pod \"network-check-target-4fmmt\" (UID: \"48523b16-8cba-40d8-bbaf-e05814fcb820\") " pod="openshift-network-diagnostics/network-check-target-4fmmt" Jul 10 18:58:45.181072 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:45.181061 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71f50128-1d81-4578-8758-cc4b92443331-metrics-certs\") pod \"network-metrics-daemon-brfp2\" (UID: \"71f50128-1d81-4578-8758-cc4b92443331\") " pod="openshift-multus/network-metrics-daemon-brfp2" Jul 10 18:58:45.181593 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:45.181094 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe-original-pull-secret\") pod \"global-pull-secret-syncer-qc4kd\" (UID: \"f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe\") " pod="kube-system/global-pull-secret-syncer-qc4kd" Jul 10 18:58:45.181593 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:45.181244 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 18:58:45.181593 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:45.181315 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe-original-pull-secret podName:f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe nodeName:}" failed. No retries permitted until 2026-07-10 18:58:53.181295496 +0000 UTC m=+17.225437980 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe-original-pull-secret") pod "global-pull-secret-syncer-qc4kd" (UID: "f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe") : object "kube-system"/"original-pull-secret" not registered Jul 10 18:58:45.181593 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:45.181348 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:58:45.181593 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:45.181439 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71f50128-1d81-4578-8758-cc4b92443331-metrics-certs podName:71f50128-1d81-4578-8758-cc4b92443331 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:53.181422165 +0000 UTC m=+17.225564647 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71f50128-1d81-4578-8758-cc4b92443331-metrics-certs") pod "network-metrics-daemon-brfp2" (UID: "71f50128-1d81-4578-8758-cc4b92443331") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:58:45.181593 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:45.181351 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 18:58:45.181593 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:45.181472 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 18:58:45.181593 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:45.181488 2581 projected.go:196] Error preparing data for projected volume kube-api-access-vwxth for pod openshift-network-diagnostics/network-check-target-4fmmt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:58:45.181593 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:45.181529 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48523b16-8cba-40d8-bbaf-e05814fcb820-kube-api-access-vwxth podName:48523b16-8cba-40d8-bbaf-e05814fcb820 nodeName:}" failed. No retries permitted until 2026-07-10 18:58:53.181516302 +0000 UTC m=+17.225658788 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vwxth" (UniqueName: "kubernetes.io/projected/48523b16-8cba-40d8-bbaf-e05814fcb820-kube-api-access-vwxth") pod "network-check-target-4fmmt" (UID: "48523b16-8cba-40d8-bbaf-e05814fcb820") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:58:45.536578 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:45.536423 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4fmmt" Jul 10 18:58:45.536578 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:45.536467 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qc4kd" Jul 10 18:58:45.536578 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:45.536423 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-brfp2" Jul 10 18:58:45.536578 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:45.536561 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4fmmt" podUID="48523b16-8cba-40d8-bbaf-e05814fcb820" Jul 10 18:58:45.536953 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:45.536669 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-brfp2" podUID="71f50128-1d81-4578-8758-cc4b92443331" Jul 10 18:58:45.536953 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:45.536774 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qc4kd" podUID="f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe" Jul 10 18:58:47.536531 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:47.536458 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-brfp2" Jul 10 18:58:47.536531 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:47.536479 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qc4kd" Jul 10 18:58:47.536531 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:47.536458 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4fmmt" Jul 10 18:58:47.537201 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:47.536605 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qc4kd" podUID="f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe" Jul 10 18:58:47.537201 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:47.536673 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4fmmt" podUID="48523b16-8cba-40d8-bbaf-e05814fcb820" Jul 10 18:58:47.537201 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:47.536797 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-brfp2" podUID="71f50128-1d81-4578-8758-cc4b92443331" Jul 10 18:58:49.536542 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:49.536513 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qc4kd" Jul 10 18:58:49.536933 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:49.536513 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-brfp2" Jul 10 18:58:49.536933 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:49.536608 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qc4kd" podUID="f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe" Jul 10 18:58:49.536933 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:49.536513 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4fmmt" Jul 10 18:58:49.536933 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:49.536701 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-brfp2" podUID="71f50128-1d81-4578-8758-cc4b92443331" Jul 10 18:58:49.536933 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:49.536740 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4fmmt" podUID="48523b16-8cba-40d8-bbaf-e05814fcb820" Jul 10 18:58:51.536345 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:51.536305 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4fmmt" Jul 10 18:58:51.536762 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:51.536362 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-brfp2" Jul 10 18:58:51.536762 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:51.536305 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qc4kd" Jul 10 18:58:51.536762 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:51.536426 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4fmmt" podUID="48523b16-8cba-40d8-bbaf-e05814fcb820" Jul 10 18:58:51.536762 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:51.536506 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-brfp2" podUID="71f50128-1d81-4578-8758-cc4b92443331" Jul 10 18:58:51.536762 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:51.536558 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qc4kd" podUID="f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe" Jul 10 18:58:53.242884 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:53.242836 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwxth\" (UniqueName: \"kubernetes.io/projected/48523b16-8cba-40d8-bbaf-e05814fcb820-kube-api-access-vwxth\") pod \"network-check-target-4fmmt\" (UID: \"48523b16-8cba-40d8-bbaf-e05814fcb820\") " pod="openshift-network-diagnostics/network-check-target-4fmmt" Jul 10 18:58:53.243347 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:53.242913 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71f50128-1d81-4578-8758-cc4b92443331-metrics-certs\") pod \"network-metrics-daemon-brfp2\" (UID: \"71f50128-1d81-4578-8758-cc4b92443331\") " pod="openshift-multus/network-metrics-daemon-brfp2" Jul 10 18:58:53.243347 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:53.242937 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe-original-pull-secret\") pod \"global-pull-secret-syncer-qc4kd\" (UID: \"f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe\") " pod="kube-system/global-pull-secret-syncer-qc4kd" Jul 10 18:58:53.243347 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:53.243024 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 18:58:53.243347 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:53.243051 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 18:58:53.243347 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:53.243058 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 18:58:53.243347 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:53.243064 2581 projected.go:196] Error preparing data for projected volume kube-api-access-vwxth for pod openshift-network-diagnostics/network-check-target-4fmmt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:58:53.243347 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:53.243059 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:58:53.243347 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:53.243120 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe-original-pull-secret podName:f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe nodeName:}" failed. No retries permitted until 2026-07-10 18:59:09.243097957 +0000 UTC m=+33.287240436 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe-original-pull-secret") pod "global-pull-secret-syncer-qc4kd" (UID: "f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe") : object "kube-system"/"original-pull-secret" not registered Jul 10 18:58:53.243347 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:53.243145 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71f50128-1d81-4578-8758-cc4b92443331-metrics-certs podName:71f50128-1d81-4578-8758-cc4b92443331 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:09.243127862 +0000 UTC m=+33.287270348 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71f50128-1d81-4578-8758-cc4b92443331-metrics-certs") pod "network-metrics-daemon-brfp2" (UID: "71f50128-1d81-4578-8758-cc4b92443331") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:58:53.243347 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:53.243166 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48523b16-8cba-40d8-bbaf-e05814fcb820-kube-api-access-vwxth podName:48523b16-8cba-40d8-bbaf-e05814fcb820 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:09.243154855 +0000 UTC m=+33.287297341 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vwxth" (UniqueName: "kubernetes.io/projected/48523b16-8cba-40d8-bbaf-e05814fcb820-kube-api-access-vwxth") pod "network-check-target-4fmmt" (UID: "48523b16-8cba-40d8-bbaf-e05814fcb820") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:58:53.536639 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:53.536550 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-brfp2" Jul 10 18:58:53.536818 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:53.536550 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qc4kd" Jul 10 18:58:53.536818 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:53.536717 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-brfp2" podUID="71f50128-1d81-4578-8758-cc4b92443331" Jul 10 18:58:53.536818 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:53.536773 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4fmmt" Jul 10 18:58:53.536956 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:53.536890 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qc4kd" podUID="f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe" Jul 10 18:58:53.537061 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:53.536983 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4fmmt" podUID="48523b16-8cba-40d8-bbaf-e05814fcb820" Jul 10 18:58:55.536635 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:55.536429 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qc4kd" Jul 10 18:58:55.537086 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:55.536458 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4fmmt" Jul 10 18:58:55.537086 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:55.536473 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-brfp2" Jul 10 18:58:55.537086 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:55.536756 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qc4kd" podUID="f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe" Jul 10 18:58:55.537086 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:55.536834 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-brfp2" podUID="71f50128-1d81-4578-8758-cc4b92443331" Jul 10 18:58:55.537086 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:55.536911 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4fmmt" podUID="48523b16-8cba-40d8-bbaf-e05814fcb820" Jul 10 18:58:57.535643 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:57.535610 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qc4kd" Jul 10 18:58:57.536013 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:57.535726 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4fmmt" Jul 10 18:58:57.536013 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:57.535732 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qc4kd" podUID="f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe" Jul 10 18:58:57.536013 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:57.535757 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-brfp2" Jul 10 18:58:57.536013 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:57.535839 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4fmmt" podUID="48523b16-8cba-40d8-bbaf-e05814fcb820" Jul 10 18:58:57.536013 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:57.535900 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-brfp2" podUID="71f50128-1d81-4578-8758-cc4b92443331" Jul 10 18:58:59.535864 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:59.535837 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qc4kd" Jul 10 18:58:59.536153 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:59.535932 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4fmmt" Jul 10 18:58:59.536153 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:59.535950 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-brfp2" Jul 10 18:58:59.536153 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:59.536008 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4fmmt" podUID="48523b16-8cba-40d8-bbaf-e05814fcb820" Jul 10 18:58:59.536153 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:59.535936 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qc4kd" podUID="f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe" Jul 10 18:58:59.536153 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:58:59.536069 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-brfp2" podUID="71f50128-1d81-4578-8758-cc4b92443331" Jul 10 18:58:59.766019 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:59.765841 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69hnr_7f479cec-2fb8-45c4-ae7d-073194a5dede/ovn-acl-logging/0.log" Jul 10 18:58:59.766377 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:59.766358 2581 generic.go:356] "Generic (PLEG): container finished" podID="7f479cec-2fb8-45c4-ae7d-073194a5dede" containerID="3a5cd4ebc86e45ce60cd3935d097020fc2a4b448fd2d552a9b39307dde49dc3c" exitCode=1 Jul 10 18:58:59.766464 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:59.766438 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" event={"ID":"7f479cec-2fb8-45c4-ae7d-073194a5dede","Type":"ContainerStarted","Data":"6234c7676fe2fad478361828e1192cd0778bfa2beba27d7dca1b3a0009356290"} Jul 10 18:58:59.766574 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:59.766531 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" event={"ID":"7f479cec-2fb8-45c4-ae7d-073194a5dede","Type":"ContainerStarted","Data":"568bc4051cf42ca669ef0b0d3ec5824fdfcd2817ba32c800308444e59b8e15f7"} Jul 10 18:58:59.766629 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:59.766588 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" event={"ID":"7f479cec-2fb8-45c4-ae7d-073194a5dede","Type":"ContainerDied","Data":"3a5cd4ebc86e45ce60cd3935d097020fc2a4b448fd2d552a9b39307dde49dc3c"} Jul 10 18:58:59.766707 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:59.766642 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" event={"ID":"7f479cec-2fb8-45c4-ae7d-073194a5dede","Type":"ContainerStarted","Data":"339be9c62cf3f74a75fcd2d0eb9512c12c1aa7ae24a72a75188fbd6b20b89c5b"} Jul 10 18:58:59.767794 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:59.767768 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gjmqz" event={"ID":"d16a97b0-104d-435b-95b9-24f19a6dbffb","Type":"ContainerStarted","Data":"01959a210f1bbe0521041d56ece8636ef4031e9cbc5686076eb1b609a319e95a"} Jul 10 18:58:59.769108 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:59.769089 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-67.ec2.internal" event={"ID":"e2729654f7488332f5966486449c5669","Type":"ContainerStarted","Data":"4bfef86231105e2269423ba8488ae9d804f6ffbaf9a8c15898a539ea2289a5d0"} Jul 10 18:58:59.770350 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:59.770331 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-k6l5w" event={"ID":"c807635a-eda8-4c7d-84bb-2f9da8c355ea","Type":"ContainerStarted","Data":"57e2e4fa28909fb9b6cda7bb0414cc40214d16621c7c5409f854b740b6f55243"} Jul 10 18:58:59.788377 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:59.788329 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-gjmqz" podStartSLOduration=1.5223305329999999 podStartE2EDuration="22.788314526s" totalImagesPullingTime="21.265983993s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:58:59.787626808 +0000 UTC m=+23.831769340" watchObservedRunningTime="2026-07-10 18:58:59.788314526 +0000 UTC m=+23.832457028" Jul 10 18:58:59.810428 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:58:59.810376 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-67.ec2.internal" podStartSLOduration=21.810355508 podStartE2EDuration="21.810355508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:58:59.810329395 +0000 UTC m=+23.854471922" watchObservedRunningTime="2026-07-10 18:58:59.810355508 +0000 UTC m=+23.854498011" Jul 10 18:59:00.775121 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:00.774931 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69hnr_7f479cec-2fb8-45c4-ae7d-073194a5dede/ovn-acl-logging/0.log" Jul 10 18:59:00.775812 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:00.775458 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" event={"ID":"7f479cec-2fb8-45c4-ae7d-073194a5dede","Type":"ContainerStarted","Data":"433c68c1cdcc9a95db01606d85b0244a92ef7800a7a52903400c9ab830638b1a"} Jul 10 18:59:00.775812 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:00.775528 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" event={"ID":"7f479cec-2fb8-45c4-ae7d-073194a5dede","Type":"ContainerStarted","Data":"c81480fc6683d6f3c5810127aeb666e537693ee57a291dfff56f010645827c75"} Jul 10 18:59:00.776883 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:00.776860 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l298r" event={"ID":"77ed8869-fdfb-453b-b2ca-c8268fd0d0aa","Type":"ContainerStarted","Data":"a75da450ebf2c2ab377aa0de9f102c8e9df426de1de4fc93b6691e50cfeaa614"} Jul 10 18:59:00.778827 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:00.778795 2581 generic.go:356] "Generic (PLEG): container finished" podID="8cb72f0a-5a49-4633-8ca4-3a4e240a4aab" containerID="8a7cdc3c2a998f6801f13e59300b2a554c03776bec751df0eea5174ee2139871" exitCode=0 Jul 10 18:59:00.778947 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:00.778869 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zxs79" event={"ID":"8cb72f0a-5a49-4633-8ca4-3a4e240a4aab","Type":"ContainerDied","Data":"8a7cdc3c2a998f6801f13e59300b2a554c03776bec751df0eea5174ee2139871"} Jul 10 18:59:00.780740 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:00.780714 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7btwf" event={"ID":"10a5a151-ab9d-45db-8502-26d00ec21659","Type":"ContainerStarted","Data":"d46df0fddcd968d74c1fdb5cdbce2c511295598f99b7270be98ea9ac5490ea4b"} Jul 10 18:59:00.782168 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:00.782141 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5z5v5" event={"ID":"7814607d-ae88-4915-bc20-345f93f50914","Type":"ContainerStarted","Data":"ed49f7e8ed340c36830324fb8a46e142860d1ba6d21429f9e66f2fb434528b47"} Jul 10 18:59:00.783380 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:00.783361 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zf82d" event={"ID":"df38c8b2-26d6-46bb-95ca-166ff4723ceb","Type":"ContainerStarted","Data":"b2ea5d60e5398fb7c93cde3338c31e48a8fe1700b04dc946d412712d951f9656"} Jul 10 18:59:00.784823 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:00.784800 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zlj4w" event={"ID":"2c842677-9d8d-4210-9a3b-ff32bf8bda1e","Type":"ContainerStarted","Data":"bb51240553d94c2d8d318de021ca2fbeaa189ebc1ebd0e2ca109a41d427d2e44"} Jul 10 18:59:00.786162 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:00.786142 2581 generic.go:356] "Generic (PLEG): container finished" podID="a2521e690c042ce0e26fc7818bf7a692" containerID="793634d749440ba20e742f7d5d9201e051970c9546f473893ff1789d95b332c7" exitCode=0 Jul 10 18:59:00.786253 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:00.786216 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-67.ec2.internal" event={"ID":"a2521e690c042ce0e26fc7818bf7a692","Type":"ContainerDied","Data":"793634d749440ba20e742f7d5d9201e051970c9546f473893ff1789d95b332c7"} Jul 10 18:59:00.801623 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:00.801588 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-k6l5w" podStartSLOduration=2.748813864 podStartE2EDuration="23.801576084s" totalImagesPullingTime="21.05276222s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:58:59.849304992 +0000 UTC m=+23.893447570" watchObservedRunningTime="2026-07-10 18:59:00.801576084 +0000 UTC m=+24.845718584" Jul 10 18:59:00.815890 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:00.815851 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zlj4w" podStartSLOduration=2.665509288 podStartE2EDuration="23.815838582s" totalImagesPullingTime="21.150329294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:00.815637745 +0000 UTC m=+24.859780245" watchObservedRunningTime="2026-07-10 18:59:00.815838582 +0000 UTC m=+24.859981083" Jul 10 18:59:00.829920 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:00.829878 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-zf82d" podStartSLOduration=2.791306812 podStartE2EDuration="23.829866735s" totalImagesPullingTime="21.038559923s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:00.829730455 +0000 UTC m=+24.873872973" watchObservedRunningTime="2026-07-10 18:59:00.829866735 +0000 UTC m=+24.874009248" Jul 10 18:59:00.844595 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:00.844546 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5z5v5" podStartSLOduration=7.527660442 podStartE2EDuration="23.844527564s" totalImagesPullingTime="16.316867122s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:00.843667521 +0000 UTC m=+24.887810049" watchObservedRunningTime="2026-07-10 18:59:00.844527564 +0000 UTC m=+24.888670066" Jul 10 18:59:00.876790 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:00.876746 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7btwf" podStartSLOduration=2.62899052 podStartE2EDuration="23.876731364s" totalImagesPullingTime="21.247740844s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:00.87631556 +0000 UTC m=+24.920458063" watchObservedRunningTime="2026-07-10 18:59:00.876731364 +0000 UTC m=+24.920873859" Jul 10 18:59:01.223980 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:01.223943 2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 10 18:59:01.494565 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:01.494418 2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-10T18:59:01.22396658Z","UUID":"5c80c101-623b-4ea2-bdde-3974ed3485a6","Handler":null,"Name":"","Endpoint":""} Jul 10 18:59:01.496067 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:01.496044 2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 10 18:59:01.496199 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:01.496088 2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 10 18:59:01.536510 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:01.536480 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qc4kd" Jul 10 18:59:01.536510 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:01.536506 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4fmmt" Jul 10 18:59:01.536733 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:01.536492 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-brfp2" Jul 10 18:59:01.536733 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:01.536603 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qc4kd" podUID="f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe" Jul 10 18:59:01.536822 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:01.536760 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-brfp2" podUID="71f50128-1d81-4578-8758-cc4b92443331" Jul 10 18:59:01.536822 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:01.536783 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4fmmt" podUID="48523b16-8cba-40d8-bbaf-e05814fcb820" Jul 10 18:59:01.791714 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:01.791659 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l298r" event={"ID":"77ed8869-fdfb-453b-b2ca-c8268fd0d0aa","Type":"ContainerStarted","Data":"68015b0c38b4226374c97a67f27328042a7f5cf8bd62d311e8d044f3eccd5b9e"} Jul 10 18:59:01.793983 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:01.793949 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-67.ec2.internal" event={"ID":"a2521e690c042ce0e26fc7818bf7a692","Type":"ContainerStarted","Data":"6cd9090f44c3d1be8fb31749a823be0ef91464396300cef0ed97f2786ef86e96"} Jul 10 18:59:01.813292 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:01.813232 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-67.ec2.internal" podStartSLOduration=23.813211334000002 podStartE2EDuration="23.813211334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:01.812371534 +0000 UTC m=+25.856514035" watchObservedRunningTime="2026-07-10 18:59:01.813211334 +0000 UTC m=+25.857353836" Jul 10 18:59:02.800310 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:02.800277 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69hnr_7f479cec-2fb8-45c4-ae7d-073194a5dede/ovn-acl-logging/0.log" Jul 10 18:59:02.800843 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:02.800671 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" event={"ID":"7f479cec-2fb8-45c4-ae7d-073194a5dede","Type":"ContainerStarted","Data":"70894b042798d2fbf915f6401fa3756dd790cece0a8d1a4a0bb5633c83bdbd31"} Jul 10 18:59:02.802990 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:02.802959 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l298r" event={"ID":"77ed8869-fdfb-453b-b2ca-c8268fd0d0aa","Type":"ContainerStarted","Data":"5dbd64ded88bd2309cf7d4df6a22b19059cd5eb0106b7cce7f4c1089d74388c5"} Jul 10 18:59:02.825599 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:02.825546 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l298r" podStartSLOduration=2.370367928 podStartE2EDuration="25.825529936s" totalImagesPullingTime="23.455162008s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:02.824669396 +0000 UTC m=+26.868811917" watchObservedRunningTime="2026-07-10 18:59:02.825529936 +0000 UTC m=+26.869672440" Jul 10 18:59:03.535730 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:03.535676 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qc4kd" Jul 10 18:59:03.535730 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:03.535732 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-brfp2" Jul 10 18:59:03.535961 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:03.535732 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4fmmt" Jul 10 18:59:03.535961 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:03.535830 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qc4kd" podUID="f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe" Jul 10 18:59:03.535961 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:03.535904 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-brfp2" podUID="71f50128-1d81-4578-8758-cc4b92443331" Jul 10 18:59:03.536116 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:03.535975 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4fmmt" podUID="48523b16-8cba-40d8-bbaf-e05814fcb820" Jul 10 18:59:05.319738 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:05.319462 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zlj4w" Jul 10 18:59:05.320453 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:05.320173 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zlj4w" Jul 10 18:59:05.535964 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:05.535924 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4fmmt" Jul 10 18:59:05.535964 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:05.535924 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qc4kd" Jul 10 18:59:05.536194 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:05.535925 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-brfp2" Jul 10 18:59:05.536194 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:05.536030 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4fmmt" podUID="48523b16-8cba-40d8-bbaf-e05814fcb820" Jul 10 18:59:05.536194 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:05.536132 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-brfp2" podUID="71f50128-1d81-4578-8758-cc4b92443331" Jul 10 18:59:05.536308 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:05.536215 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qc4kd" podUID="f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe" Jul 10 18:59:05.813352 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:05.813326 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69hnr_7f479cec-2fb8-45c4-ae7d-073194a5dede/ovn-acl-logging/0.log" Jul 10 18:59:05.813769 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:05.813749 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" event={"ID":"7f479cec-2fb8-45c4-ae7d-073194a5dede","Type":"ContainerStarted","Data":"d2e2c25d9e44e5983843ecff20ce176f9c751c28a3e8a5a63f41d733a1ed2146"} Jul 10 18:59:05.814061 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:05.814038 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:59:05.814169 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:05.814111 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:59:05.814169 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:05.814154 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:59:05.814248 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:05.814193 2581 scope.go:117] "RemoveContainer" containerID="3a5cd4ebc86e45ce60cd3935d097020fc2a4b448fd2d552a9b39307dde49dc3c" Jul 10 18:59:05.815790 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:05.815766 2581 generic.go:356] "Generic (PLEG): container finished" podID="8cb72f0a-5a49-4633-8ca4-3a4e240a4aab" containerID="5ef7226ae9969b745f9a29ccf4ffa3cd775690d05619ab105bb83ee1975c64bc" exitCode=0 Jul 10 18:59:05.815886 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:05.815849 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zxs79" event={"ID":"8cb72f0a-5a49-4633-8ca4-3a4e240a4aab","Type":"ContainerDied","Data":"5ef7226ae9969b745f9a29ccf4ffa3cd775690d05619ab105bb83ee1975c64bc"} Jul 10 18:59:05.816239 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:05.816218 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zlj4w" Jul 10 18:59:05.816738 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:05.816615 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zlj4w" Jul 10 18:59:05.830986 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:05.830965 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:59:05.831077 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:05.831066 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:59:06.820725 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:06.820678 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zxs79" event={"ID":"8cb72f0a-5a49-4633-8ca4-3a4e240a4aab","Type":"ContainerStarted","Data":"1ffbcd314b59ad4bc680b84411c5e8d0276406f3dd9c2f7ed89944b0bf21dc23"} Jul 10 18:59:06.824204 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:06.824189 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69hnr_7f479cec-2fb8-45c4-ae7d-073194a5dede/ovn-acl-logging/0.log" Jul 10 18:59:06.824553 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:06.824531 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" event={"ID":"7f479cec-2fb8-45c4-ae7d-073194a5dede","Type":"ContainerStarted","Data":"d25f6924079103c58ce53d4c03399fe661a461a24a5a65b353f6c630db4a3f5c"} Jul 10 18:59:06.895498 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:06.895464 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-brfp2"] Jul 10 18:59:06.895657 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:06.895581 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-brfp2" Jul 10 18:59:06.895734 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:06.895707 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-brfp2" podUID="71f50128-1d81-4578-8758-cc4b92443331" Jul 10 18:59:06.896113 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:06.896089 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qc4kd"] Jul 10 18:59:06.896199 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:06.896178 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qc4kd" Jul 10 18:59:06.896270 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:06.896253 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qc4kd" podUID="f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe" Jul 10 18:59:06.901033 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:06.901006 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4fmmt"] Jul 10 18:59:06.901139 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:06.901111 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4fmmt" Jul 10 18:59:06.901214 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:06.901197 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4fmmt" podUID="48523b16-8cba-40d8-bbaf-e05814fcb820" Jul 10 18:59:06.928843 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:06.928727 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" podStartSLOduration=8.562121651 podStartE2EDuration="29.928708888s" totalImagesPullingTime="21.366587237s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:06.928012982 +0000 UTC m=+30.972155494" watchObservedRunningTime="2026-07-10 18:59:06.928708888 +0000 UTC m=+30.972851389" Jul 10 18:59:07.828959 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:07.828925 2581 generic.go:356] "Generic (PLEG): container finished" podID="8cb72f0a-5a49-4633-8ca4-3a4e240a4aab" containerID="1ffbcd314b59ad4bc680b84411c5e8d0276406f3dd9c2f7ed89944b0bf21dc23" exitCode=0 Jul 10 18:59:07.829439 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:07.829014 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zxs79" event={"ID":"8cb72f0a-5a49-4633-8ca4-3a4e240a4aab","Type":"ContainerDied","Data":"1ffbcd314b59ad4bc680b84411c5e8d0276406f3dd9c2f7ed89944b0bf21dc23"} Jul 10 18:59:08.536229 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:08.536188 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-brfp2" Jul 10 18:59:08.536414 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:08.536188 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4fmmt" Jul 10 18:59:08.536414 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:08.536208 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qc4kd" Jul 10 18:59:08.536414 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:08.536332 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-brfp2" podUID="71f50128-1d81-4578-8758-cc4b92443331" Jul 10 18:59:08.536550 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:08.536433 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4fmmt" podUID="48523b16-8cba-40d8-bbaf-e05814fcb820" Jul 10 18:59:08.536550 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:08.536519 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qc4kd" podUID="f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe" Jul 10 18:59:09.269550 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:09.269518 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwxth\" (UniqueName: \"kubernetes.io/projected/48523b16-8cba-40d8-bbaf-e05814fcb820-kube-api-access-vwxth\") pod \"network-check-target-4fmmt\" (UID: \"48523b16-8cba-40d8-bbaf-e05814fcb820\") " pod="openshift-network-diagnostics/network-check-target-4fmmt" Jul 10 18:59:09.269937 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:09.269574 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71f50128-1d81-4578-8758-cc4b92443331-metrics-certs\") pod \"network-metrics-daemon-brfp2\" (UID: \"71f50128-1d81-4578-8758-cc4b92443331\") " pod="openshift-multus/network-metrics-daemon-brfp2" Jul 10 18:59:09.269937 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:09.269600 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe-original-pull-secret\") pod \"global-pull-secret-syncer-qc4kd\" (UID: \"f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe\") " pod="kube-system/global-pull-secret-syncer-qc4kd" Jul 10 18:59:09.269937 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:09.269704 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 18:59:09.269937 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:09.269715 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 18:59:09.269937 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:09.269731 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 18:59:09.269937 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:09.269742 2581 projected.go:196] Error preparing data for projected volume kube-api-access-vwxth for pod openshift-network-diagnostics/network-check-target-4fmmt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:59:09.269937 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:09.269748 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:59:09.269937 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:09.269767 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe-original-pull-secret podName:f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe nodeName:}" failed. No retries permitted until 2026-07-10 18:59:41.269749083 +0000 UTC m=+65.313891564 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe-original-pull-secret") pod "global-pull-secret-syncer-qc4kd" (UID: "f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe") : object "kube-system"/"original-pull-secret" not registered Jul 10 18:59:09.269937 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:09.269786 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48523b16-8cba-40d8-bbaf-e05814fcb820-kube-api-access-vwxth podName:48523b16-8cba-40d8-bbaf-e05814fcb820 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:41.269776751 +0000 UTC m=+65.313919234 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-vwxth" (UniqueName: "kubernetes.io/projected/48523b16-8cba-40d8-bbaf-e05814fcb820-kube-api-access-vwxth") pod "network-check-target-4fmmt" (UID: "48523b16-8cba-40d8-bbaf-e05814fcb820") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 18:59:09.269937 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:09.269799 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71f50128-1d81-4578-8758-cc4b92443331-metrics-certs podName:71f50128-1d81-4578-8758-cc4b92443331 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:41.269793165 +0000 UTC m=+65.313935645 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71f50128-1d81-4578-8758-cc4b92443331-metrics-certs") pod "network-metrics-daemon-brfp2" (UID: "71f50128-1d81-4578-8758-cc4b92443331") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 18:59:09.835574 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:09.835547 2581 generic.go:356] "Generic (PLEG): container finished" podID="8cb72f0a-5a49-4633-8ca4-3a4e240a4aab" containerID="e1980e5656e3d100a49f073423a0e192c1a33fac7a60e0441864bd1af84c9938" exitCode=0 Jul 10 18:59:09.835746 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:09.835590 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zxs79" event={"ID":"8cb72f0a-5a49-4633-8ca4-3a4e240a4aab","Type":"ContainerDied","Data":"e1980e5656e3d100a49f073423a0e192c1a33fac7a60e0441864bd1af84c9938"} Jul 10 18:59:10.535965 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:10.535932 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4fmmt" Jul 10 18:59:10.536436 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:10.536064 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4fmmt" podUID="48523b16-8cba-40d8-bbaf-e05814fcb820" Jul 10 18:59:10.536436 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:10.536115 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qc4kd" Jul 10 18:59:10.536436 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:10.536197 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qc4kd" podUID="f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe" Jul 10 18:59:10.536436 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:10.535938 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-brfp2" Jul 10 18:59:10.536436 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:10.536283 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-brfp2" podUID="71f50128-1d81-4578-8758-cc4b92443331" Jul 10 18:59:12.537257 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:12.536573 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-brfp2" Jul 10 18:59:12.537257 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:12.537070 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-brfp2" podUID="71f50128-1d81-4578-8758-cc4b92443331" Jul 10 18:59:12.537257 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:12.537112 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4fmmt" Jul 10 18:59:12.537257 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:12.537115 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qc4kd" Jul 10 18:59:12.537257 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:12.537237 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4fmmt" podUID="48523b16-8cba-40d8-bbaf-e05814fcb820" Jul 10 18:59:12.537908 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:12.537302 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qc4kd" podUID="f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe" Jul 10 18:59:12.795587 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:12.795517 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-67.ec2.internal" event="NodeReady" Jul 10 18:59:12.795760 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:12.795655 2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 10 18:59:12.842845 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:12.842814 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-v5rql"] Jul 10 18:59:12.848050 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:12.848024 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-szfx9"] Jul 10 18:59:12.848197 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:12.848158 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-v5rql" Jul 10 18:59:12.850852 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:12.850828 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-s6z54\"" Jul 10 18:59:12.850965 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:12.850922 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 10 18:59:12.850965 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:12.850939 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 10 18:59:12.851699 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:12.851666 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-szfx9" Jul 10 18:59:12.854001 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:12.853981 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 10 18:59:12.854001 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:12.854005 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-5z57c\"" Jul 10 18:59:12.854189 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:12.853981 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 10 18:59:12.854189 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:12.854065 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 10 18:59:12.856130 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:12.856111 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-v5rql"] Jul 10 18:59:12.860207 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:12.860185 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-szfx9"] Jul 10 18:59:12.994162 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:12.994124 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ed9ac220-276d-4864-b19b-17b8f69eaa98-metrics-tls\") pod \"dns-default-v5rql\" (UID: \"ed9ac220-276d-4864-b19b-17b8f69eaa98\") " pod="openshift-dns/dns-default-v5rql" Jul 10 18:59:12.994321 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:12.994168 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3a2ad480-1b70-460f-8fe5-e0dafed541e5-cert\") pod \"ingress-canary-szfx9\" (UID: \"3a2ad480-1b70-460f-8fe5-e0dafed541e5\") " pod="openshift-ingress-canary/ingress-canary-szfx9" Jul 10 18:59:12.994321 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:12.994212 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vtss\" (UniqueName: \"kubernetes.io/projected/3a2ad480-1b70-460f-8fe5-e0dafed541e5-kube-api-access-4vtss\") pod \"ingress-canary-szfx9\" (UID: \"3a2ad480-1b70-460f-8fe5-e0dafed541e5\") " pod="openshift-ingress-canary/ingress-canary-szfx9" Jul 10 18:59:12.994321 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:12.994249 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r89zk\" (UniqueName: \"kubernetes.io/projected/ed9ac220-276d-4864-b19b-17b8f69eaa98-kube-api-access-r89zk\") pod \"dns-default-v5rql\" (UID: \"ed9ac220-276d-4864-b19b-17b8f69eaa98\") " pod="openshift-dns/dns-default-v5rql" Jul 10 18:59:12.994321 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:12.994306 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ed9ac220-276d-4864-b19b-17b8f69eaa98-config-volume\") pod \"dns-default-v5rql\" (UID: \"ed9ac220-276d-4864-b19b-17b8f69eaa98\") " pod="openshift-dns/dns-default-v5rql" Jul 10 18:59:12.994490 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:12.994338 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed9ac220-276d-4864-b19b-17b8f69eaa98-tmp-dir\") pod \"dns-default-v5rql\" (UID: \"ed9ac220-276d-4864-b19b-17b8f69eaa98\") " pod="openshift-dns/dns-default-v5rql" Jul 10 18:59:13.095402 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:13.095318 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ed9ac220-276d-4864-b19b-17b8f69eaa98-metrics-tls\") pod \"dns-default-v5rql\" (UID: \"ed9ac220-276d-4864-b19b-17b8f69eaa98\") " pod="openshift-dns/dns-default-v5rql" Jul 10 18:59:13.095402 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:13.095358 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3a2ad480-1b70-460f-8fe5-e0dafed541e5-cert\") pod \"ingress-canary-szfx9\" (UID: \"3a2ad480-1b70-460f-8fe5-e0dafed541e5\") " pod="openshift-ingress-canary/ingress-canary-szfx9" Jul 10 18:59:13.095402 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:13.095397 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vtss\" (UniqueName: \"kubernetes.io/projected/3a2ad480-1b70-460f-8fe5-e0dafed541e5-kube-api-access-4vtss\") pod \"ingress-canary-szfx9\" (UID: \"3a2ad480-1b70-460f-8fe5-e0dafed541e5\") " pod="openshift-ingress-canary/ingress-canary-szfx9" Jul 10 18:59:13.095702 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:13.095440 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r89zk\" (UniqueName: \"kubernetes.io/projected/ed9ac220-276d-4864-b19b-17b8f69eaa98-kube-api-access-r89zk\") pod \"dns-default-v5rql\" (UID: \"ed9ac220-276d-4864-b19b-17b8f69eaa98\") " pod="openshift-dns/dns-default-v5rql" Jul 10 18:59:13.095702 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:13.095488 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ed9ac220-276d-4864-b19b-17b8f69eaa98-config-volume\") pod \"dns-default-v5rql\" (UID: \"ed9ac220-276d-4864-b19b-17b8f69eaa98\") " pod="openshift-dns/dns-default-v5rql" Jul 10 18:59:13.095702 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:13.095499 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 18:59:13.095702 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:13.095510 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed9ac220-276d-4864-b19b-17b8f69eaa98-tmp-dir\") pod \"dns-default-v5rql\" (UID: \"ed9ac220-276d-4864-b19b-17b8f69eaa98\") " pod="openshift-dns/dns-default-v5rql" Jul 10 18:59:13.095702 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:13.095572 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed9ac220-276d-4864-b19b-17b8f69eaa98-metrics-tls podName:ed9ac220-276d-4864-b19b-17b8f69eaa98 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:13.595551947 +0000 UTC m=+37.639694451 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ed9ac220-276d-4864-b19b-17b8f69eaa98-metrics-tls") pod "dns-default-v5rql" (UID: "ed9ac220-276d-4864-b19b-17b8f69eaa98") : secret "dns-default-metrics-tls" not found Jul 10 18:59:13.095878 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:13.095850 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed9ac220-276d-4864-b19b-17b8f69eaa98-tmp-dir\") pod \"dns-default-v5rql\" (UID: \"ed9ac220-276d-4864-b19b-17b8f69eaa98\") " pod="openshift-dns/dns-default-v5rql" Jul 10 18:59:13.096104 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:13.096085 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ed9ac220-276d-4864-b19b-17b8f69eaa98-config-volume\") pod \"dns-default-v5rql\" (UID: \"ed9ac220-276d-4864-b19b-17b8f69eaa98\") " pod="openshift-dns/dns-default-v5rql" Jul 10 18:59:13.100339 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:13.100314 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3a2ad480-1b70-460f-8fe5-e0dafed541e5-cert\") pod \"ingress-canary-szfx9\" (UID: \"3a2ad480-1b70-460f-8fe5-e0dafed541e5\") " pod="openshift-ingress-canary/ingress-canary-szfx9" Jul 10 18:59:13.114444 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:13.114415 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vtss\" (UniqueName: \"kubernetes.io/projected/3a2ad480-1b70-460f-8fe5-e0dafed541e5-kube-api-access-4vtss\") pod \"ingress-canary-szfx9\" (UID: \"3a2ad480-1b70-460f-8fe5-e0dafed541e5\") " pod="openshift-ingress-canary/ingress-canary-szfx9" Jul 10 18:59:13.115071 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:13.115048 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r89zk\" (UniqueName: \"kubernetes.io/projected/ed9ac220-276d-4864-b19b-17b8f69eaa98-kube-api-access-r89zk\") pod \"dns-default-v5rql\" (UID: \"ed9ac220-276d-4864-b19b-17b8f69eaa98\") " pod="openshift-dns/dns-default-v5rql" Jul 10 18:59:13.168033 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:13.167995 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-szfx9" Jul 10 18:59:13.598786 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:13.598746 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ed9ac220-276d-4864-b19b-17b8f69eaa98-metrics-tls\") pod \"dns-default-v5rql\" (UID: \"ed9ac220-276d-4864-b19b-17b8f69eaa98\") " pod="openshift-dns/dns-default-v5rql" Jul 10 18:59:13.599275 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:13.598885 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 18:59:13.599275 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:13.598951 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed9ac220-276d-4864-b19b-17b8f69eaa98-metrics-tls podName:ed9ac220-276d-4864-b19b-17b8f69eaa98 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:14.598934802 +0000 UTC m=+38.643077280 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ed9ac220-276d-4864-b19b-17b8f69eaa98-metrics-tls") pod "dns-default-v5rql" (UID: "ed9ac220-276d-4864-b19b-17b8f69eaa98") : secret "dns-default-metrics-tls" not found Jul 10 18:59:13.838470 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:13.838378 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-szfx9"] Jul 10 18:59:13.841380 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:59:13.841168 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a2ad480_1b70_460f_8fe5_e0dafed541e5.slice/crio-eba840ce08917c39aaeea07e234f3c5f70cd95a782e241a6a67e27ae3a9df192 WatchSource:0}: Error finding container eba840ce08917c39aaeea07e234f3c5f70cd95a782e241a6a67e27ae3a9df192: Status 404 returned error can't find the container with id eba840ce08917c39aaeea07e234f3c5f70cd95a782e241a6a67e27ae3a9df192 Jul 10 18:59:13.845742 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:13.845717 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-szfx9" event={"ID":"3a2ad480-1b70-460f-8fe5-e0dafed541e5","Type":"ContainerStarted","Data":"eba840ce08917c39aaeea07e234f3c5f70cd95a782e241a6a67e27ae3a9df192"} Jul 10 18:59:14.536532 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:14.536492 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qc4kd" Jul 10 18:59:14.536532 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:14.536528 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4fmmt" Jul 10 18:59:14.536806 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:14.536496 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-brfp2" Jul 10 18:59:14.540946 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:14.540922 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xwgdk\"" Jul 10 18:59:14.540946 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:14.540938 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 18:59:14.541128 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:14.540974 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mzc76\"" Jul 10 18:59:14.541128 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:14.540922 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 18:59:14.541232 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:14.541179 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 18:59:14.541290 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:14.541240 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 18:59:14.606904 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:14.606874 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ed9ac220-276d-4864-b19b-17b8f69eaa98-metrics-tls\") pod \"dns-default-v5rql\" (UID: \"ed9ac220-276d-4864-b19b-17b8f69eaa98\") " pod="openshift-dns/dns-default-v5rql" Jul 10 18:59:14.607482 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:14.606988 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 18:59:14.607482 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:14.607050 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed9ac220-276d-4864-b19b-17b8f69eaa98-metrics-tls podName:ed9ac220-276d-4864-b19b-17b8f69eaa98 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:16.607031301 +0000 UTC m=+40.651173782 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ed9ac220-276d-4864-b19b-17b8f69eaa98-metrics-tls") pod "dns-default-v5rql" (UID: "ed9ac220-276d-4864-b19b-17b8f69eaa98") : secret "dns-default-metrics-tls" not found Jul 10 18:59:14.851466 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:14.851337 2581 generic.go:356] "Generic (PLEG): container finished" podID="8cb72f0a-5a49-4633-8ca4-3a4e240a4aab" containerID="ea56f2ce7517d2f1c04874500e3a3e2a363d7e601590bf34471ab75b7b31195f" exitCode=0 Jul 10 18:59:14.851466 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:14.851404 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zxs79" event={"ID":"8cb72f0a-5a49-4633-8ca4-3a4e240a4aab","Type":"ContainerDied","Data":"ea56f2ce7517d2f1c04874500e3a3e2a363d7e601590bf34471ab75b7b31195f"} Jul 10 18:59:15.858537 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:15.858491 2581 generic.go:356] "Generic (PLEG): container finished" podID="8cb72f0a-5a49-4633-8ca4-3a4e240a4aab" containerID="00085bebc95e71838e99fcd9517ab86c20a19f99ae607408f980b7ecbc0751f5" exitCode=0 Jul 10 18:59:15.858537 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:15.858527 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zxs79" event={"ID":"8cb72f0a-5a49-4633-8ca4-3a4e240a4aab","Type":"ContainerDied","Data":"00085bebc95e71838e99fcd9517ab86c20a19f99ae607408f980b7ecbc0751f5"} Jul 10 18:59:16.626379 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:16.626335 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ed9ac220-276d-4864-b19b-17b8f69eaa98-metrics-tls\") pod \"dns-default-v5rql\" (UID: \"ed9ac220-276d-4864-b19b-17b8f69eaa98\") " pod="openshift-dns/dns-default-v5rql" Jul 10 18:59:16.626548 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:16.626484 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 18:59:16.626586 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:16.626552 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed9ac220-276d-4864-b19b-17b8f69eaa98-metrics-tls podName:ed9ac220-276d-4864-b19b-17b8f69eaa98 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:20.626536476 +0000 UTC m=+44.670678954 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ed9ac220-276d-4864-b19b-17b8f69eaa98-metrics-tls") pod "dns-default-v5rql" (UID: "ed9ac220-276d-4864-b19b-17b8f69eaa98") : secret "dns-default-metrics-tls" not found Jul 10 18:59:16.864574 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:16.864355 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zxs79" event={"ID":"8cb72f0a-5a49-4633-8ca4-3a4e240a4aab","Type":"ContainerStarted","Data":"f371ede29ac168c53bbd31339e36bf43a88eebd36ee3590dd570fb8f1a20693b"} Jul 10 18:59:16.865964 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:16.865935 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-szfx9" event={"ID":"3a2ad480-1b70-460f-8fe5-e0dafed541e5","Type":"ContainerStarted","Data":"ba17777cc6bbc76ad9adc08dbd8fe2b59925139e49b3c80bd510880400a286b4"} Jul 10 18:59:16.901562 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:16.901506 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zxs79" podStartSLOduration=8.635074025 podStartE2EDuration="39.901489405s" totalImagesPullingTime="31.26641538s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:37 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:16.900782657 +0000 UTC m=+40.944925174" watchObservedRunningTime="2026-07-10 18:59:16.901489405 +0000 UTC m=+40.945631906" Jul 10 18:59:16.932209 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:16.932149 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-szfx9" podStartSLOduration=2.387820661 podStartE2EDuration="4.932133872s" totalImagesPullingTime="2.544313211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:59:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:16.930045268 +0000 UTC m=+40.974187769" watchObservedRunningTime="2026-07-10 18:59:16.932133872 +0000 UTC m=+40.976276373" Jul 10 18:59:20.653874 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:20.653836 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ed9ac220-276d-4864-b19b-17b8f69eaa98-metrics-tls\") pod \"dns-default-v5rql\" (UID: \"ed9ac220-276d-4864-b19b-17b8f69eaa98\") " pod="openshift-dns/dns-default-v5rql" Jul 10 18:59:20.654355 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:20.653986 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 18:59:20.654355 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:20.654065 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed9ac220-276d-4864-b19b-17b8f69eaa98-metrics-tls podName:ed9ac220-276d-4864-b19b-17b8f69eaa98 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:28.654045889 +0000 UTC m=+52.698188368 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ed9ac220-276d-4864-b19b-17b8f69eaa98-metrics-tls") pod "dns-default-v5rql" (UID: "ed9ac220-276d-4864-b19b-17b8f69eaa98") : secret "dns-default-metrics-tls" not found Jul 10 18:59:28.705556 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:28.705513 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ed9ac220-276d-4864-b19b-17b8f69eaa98-metrics-tls\") pod \"dns-default-v5rql\" (UID: \"ed9ac220-276d-4864-b19b-17b8f69eaa98\") " pod="openshift-dns/dns-default-v5rql" Jul 10 18:59:28.706055 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:28.705672 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 18:59:28.706055 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:28.705756 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed9ac220-276d-4864-b19b-17b8f69eaa98-metrics-tls podName:ed9ac220-276d-4864-b19b-17b8f69eaa98 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:44.70573861 +0000 UTC m=+68.749881090 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ed9ac220-276d-4864-b19b-17b8f69eaa98-metrics-tls") pod "dns-default-v5rql" (UID: "ed9ac220-276d-4864-b19b-17b8f69eaa98") : secret "dns-default-metrics-tls" not found Jul 10 18:59:37.842138 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:37.842111 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-69hnr" Jul 10 18:59:38.367535 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:38.367502 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-68z78"] Jul 10 18:59:38.388088 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:38.388065 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-8c8d8cb74-pbqx5"] Jul 10 18:59:38.388238 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:38.388192 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-68z78" Jul 10 18:59:38.395600 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:38.395574 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-j798f\"" Jul 10 18:59:38.395727 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:38.395614 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 10 18:59:38.395727 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:38.395628 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 10 18:59:38.402108 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:38.402084 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-68z78"] Jul 10 18:59:38.402229 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:38.402190 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-8c8d8cb74-pbqx5" Jul 10 18:59:38.409060 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:38.409040 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-8c8d8cb74-pbqx5"] Jul 10 18:59:38.409354 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:38.409338 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 10 18:59:38.412191 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:38.412173 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-mpncv\"" Jul 10 18:59:38.413045 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:38.413014 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 10 18:59:38.413045 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:38.413029 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 10 18:59:38.413190 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:38.413097 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 10 18:59:38.413247 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:38.413016 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 10 18:59:38.413247 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:38.413237 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 10 18:59:38.476176 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:38.476142 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/348b64e9-4db6-4d74-9a29-36436b02e1d3-service-ca-bundle\") pod \"router-default-8c8d8cb74-pbqx5\" (UID: \"348b64e9-4db6-4d74-9a29-36436b02e1d3\") " pod="openshift-ingress/router-default-8c8d8cb74-pbqx5" Jul 10 18:59:38.476355 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:38.476197 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/348b64e9-4db6-4d74-9a29-36436b02e1d3-metrics-certs\") pod \"router-default-8c8d8cb74-pbqx5\" (UID: \"348b64e9-4db6-4d74-9a29-36436b02e1d3\") " pod="openshift-ingress/router-default-8c8d8cb74-pbqx5" Jul 10 18:59:38.476355 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:38.476291 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/348b64e9-4db6-4d74-9a29-36436b02e1d3-stats-auth\") pod \"router-default-8c8d8cb74-pbqx5\" (UID: \"348b64e9-4db6-4d74-9a29-36436b02e1d3\") " pod="openshift-ingress/router-default-8c8d8cb74-pbqx5" Jul 10 18:59:38.476355 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:38.476317 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2jn8\" (UniqueName: \"kubernetes.io/projected/348b64e9-4db6-4d74-9a29-36436b02e1d3-kube-api-access-b2jn8\") pod \"router-default-8c8d8cb74-pbqx5\" (UID: \"348b64e9-4db6-4d74-9a29-36436b02e1d3\") " pod="openshift-ingress/router-default-8c8d8cb74-pbqx5" Jul 10 18:59:38.476512 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:38.476388 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/348b64e9-4db6-4d74-9a29-36436b02e1d3-default-certificate\") pod \"router-default-8c8d8cb74-pbqx5\" (UID: \"348b64e9-4db6-4d74-9a29-36436b02e1d3\") " pod="openshift-ingress/router-default-8c8d8cb74-pbqx5" Jul 10 18:59:38.476512 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:38.476430 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqwpx\" (UniqueName: \"kubernetes.io/projected/1f90ac19-a36e-4ee5-ba73-c7f06f7950f9-kube-api-access-mqwpx\") pod \"volume-data-source-validator-78495597f6-68z78\" (UID: \"1f90ac19-a36e-4ee5-ba73-c7f06f7950f9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-68z78" Jul 10 18:59:38.577038 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:38.577005 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/348b64e9-4db6-4d74-9a29-36436b02e1d3-service-ca-bundle\") pod \"router-default-8c8d8cb74-pbqx5\" (UID: \"348b64e9-4db6-4d74-9a29-36436b02e1d3\") " pod="openshift-ingress/router-default-8c8d8cb74-pbqx5" Jul 10 18:59:38.577203 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:38.577056 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/348b64e9-4db6-4d74-9a29-36436b02e1d3-metrics-certs\") pod \"router-default-8c8d8cb74-pbqx5\" (UID: \"348b64e9-4db6-4d74-9a29-36436b02e1d3\") " pod="openshift-ingress/router-default-8c8d8cb74-pbqx5" Jul 10 18:59:38.577203 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:38.577164 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 18:59:38.577203 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:38.577193 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/348b64e9-4db6-4d74-9a29-36436b02e1d3-service-ca-bundle podName:348b64e9-4db6-4d74-9a29-36436b02e1d3 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:39.07717459 +0000 UTC m=+63.121317072 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/348b64e9-4db6-4d74-9a29-36436b02e1d3-service-ca-bundle") pod "router-default-8c8d8cb74-pbqx5" (UID: "348b64e9-4db6-4d74-9a29-36436b02e1d3") : configmap references non-existent config key: service-ca.crt Jul 10 18:59:38.577363 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:38.577215 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/348b64e9-4db6-4d74-9a29-36436b02e1d3-metrics-certs podName:348b64e9-4db6-4d74-9a29-36436b02e1d3 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:39.077208064 +0000 UTC m=+63.121350543 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/348b64e9-4db6-4d74-9a29-36436b02e1d3-metrics-certs") pod "router-default-8c8d8cb74-pbqx5" (UID: "348b64e9-4db6-4d74-9a29-36436b02e1d3") : secret "router-metrics-certs-default" not found Jul 10 18:59:38.577363 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:38.577241 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/348b64e9-4db6-4d74-9a29-36436b02e1d3-stats-auth\") pod \"router-default-8c8d8cb74-pbqx5\" (UID: \"348b64e9-4db6-4d74-9a29-36436b02e1d3\") " pod="openshift-ingress/router-default-8c8d8cb74-pbqx5" Jul 10 18:59:38.577363 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:38.577265 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2jn8\" (UniqueName: \"kubernetes.io/projected/348b64e9-4db6-4d74-9a29-36436b02e1d3-kube-api-access-b2jn8\") pod \"router-default-8c8d8cb74-pbqx5\" (UID: \"348b64e9-4db6-4d74-9a29-36436b02e1d3\") " pod="openshift-ingress/router-default-8c8d8cb74-pbqx5" Jul 10 18:59:38.577363 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:38.577289 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/348b64e9-4db6-4d74-9a29-36436b02e1d3-default-certificate\") pod \"router-default-8c8d8cb74-pbqx5\" (UID: \"348b64e9-4db6-4d74-9a29-36436b02e1d3\") " pod="openshift-ingress/router-default-8c8d8cb74-pbqx5" Jul 10 18:59:38.577363 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:38.577316 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqwpx\" (UniqueName: \"kubernetes.io/projected/1f90ac19-a36e-4ee5-ba73-c7f06f7950f9-kube-api-access-mqwpx\") pod \"volume-data-source-validator-78495597f6-68z78\" (UID: \"1f90ac19-a36e-4ee5-ba73-c7f06f7950f9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-68z78" Jul 10 18:59:38.581144 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:38.581110 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/348b64e9-4db6-4d74-9a29-36436b02e1d3-default-certificate\") pod \"router-default-8c8d8cb74-pbqx5\" (UID: \"348b64e9-4db6-4d74-9a29-36436b02e1d3\") " pod="openshift-ingress/router-default-8c8d8cb74-pbqx5" Jul 10 18:59:38.581144 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:38.581135 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/348b64e9-4db6-4d74-9a29-36436b02e1d3-stats-auth\") pod \"router-default-8c8d8cb74-pbqx5\" (UID: \"348b64e9-4db6-4d74-9a29-36436b02e1d3\") " pod="openshift-ingress/router-default-8c8d8cb74-pbqx5" Jul 10 18:59:38.592813 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:38.592789 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqwpx\" (UniqueName: \"kubernetes.io/projected/1f90ac19-a36e-4ee5-ba73-c7f06f7950f9-kube-api-access-mqwpx\") pod \"volume-data-source-validator-78495597f6-68z78\" (UID: \"1f90ac19-a36e-4ee5-ba73-c7f06f7950f9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-68z78" Jul 10 18:59:38.594063 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:38.594047 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2jn8\" (UniqueName: \"kubernetes.io/projected/348b64e9-4db6-4d74-9a29-36436b02e1d3-kube-api-access-b2jn8\") pod \"router-default-8c8d8cb74-pbqx5\" (UID: \"348b64e9-4db6-4d74-9a29-36436b02e1d3\") " pod="openshift-ingress/router-default-8c8d8cb74-pbqx5" Jul 10 18:59:38.697369 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:38.697283 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-68z78" Jul 10 18:59:38.849795 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:38.849761 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-68z78"] Jul 10 18:59:38.853180 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:59:38.853151 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f90ac19_a36e_4ee5_ba73_c7f06f7950f9.slice/crio-448e88ea76133e746ecd03e75a4dfe674b8e14b7be38e79430d0dd0f77785a29 WatchSource:0}: Error finding container 448e88ea76133e746ecd03e75a4dfe674b8e14b7be38e79430d0dd0f77785a29: Status 404 returned error can't find the container with id 448e88ea76133e746ecd03e75a4dfe674b8e14b7be38e79430d0dd0f77785a29 Jul 10 18:59:38.923376 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:38.923344 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-68z78" event={"ID":"1f90ac19-a36e-4ee5-ba73-c7f06f7950f9","Type":"ContainerStarted","Data":"448e88ea76133e746ecd03e75a4dfe674b8e14b7be38e79430d0dd0f77785a29"} Jul 10 18:59:39.081349 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:39.081314 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/348b64e9-4db6-4d74-9a29-36436b02e1d3-service-ca-bundle\") pod \"router-default-8c8d8cb74-pbqx5\" (UID: \"348b64e9-4db6-4d74-9a29-36436b02e1d3\") " pod="openshift-ingress/router-default-8c8d8cb74-pbqx5" Jul 10 18:59:39.081546 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:39.081367 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/348b64e9-4db6-4d74-9a29-36436b02e1d3-metrics-certs\") pod \"router-default-8c8d8cb74-pbqx5\" (UID: \"348b64e9-4db6-4d74-9a29-36436b02e1d3\") " pod="openshift-ingress/router-default-8c8d8cb74-pbqx5" Jul 10 18:59:39.081546 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:39.081463 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 18:59:39.081546 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:39.081505 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/348b64e9-4db6-4d74-9a29-36436b02e1d3-service-ca-bundle podName:348b64e9-4db6-4d74-9a29-36436b02e1d3 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:40.081481313 +0000 UTC m=+64.125623792 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/348b64e9-4db6-4d74-9a29-36436b02e1d3-service-ca-bundle") pod "router-default-8c8d8cb74-pbqx5" (UID: "348b64e9-4db6-4d74-9a29-36436b02e1d3") : configmap references non-existent config key: service-ca.crt Jul 10 18:59:39.081546 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:39.081534 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/348b64e9-4db6-4d74-9a29-36436b02e1d3-metrics-certs podName:348b64e9-4db6-4d74-9a29-36436b02e1d3 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:40.081525954 +0000 UTC m=+64.125668435 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/348b64e9-4db6-4d74-9a29-36436b02e1d3-metrics-certs") pod "router-default-8c8d8cb74-pbqx5" (UID: "348b64e9-4db6-4d74-9a29-36436b02e1d3") : secret "router-metrics-certs-default" not found Jul 10 18:59:40.089630 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:40.089599 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/348b64e9-4db6-4d74-9a29-36436b02e1d3-service-ca-bundle\") pod \"router-default-8c8d8cb74-pbqx5\" (UID: \"348b64e9-4db6-4d74-9a29-36436b02e1d3\") " pod="openshift-ingress/router-default-8c8d8cb74-pbqx5" Jul 10 18:59:40.090070 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:40.089643 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/348b64e9-4db6-4d74-9a29-36436b02e1d3-metrics-certs\") pod \"router-default-8c8d8cb74-pbqx5\" (UID: \"348b64e9-4db6-4d74-9a29-36436b02e1d3\") " pod="openshift-ingress/router-default-8c8d8cb74-pbqx5" Jul 10 18:59:40.090070 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:40.089789 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/348b64e9-4db6-4d74-9a29-36436b02e1d3-service-ca-bundle podName:348b64e9-4db6-4d74-9a29-36436b02e1d3 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:42.089768678 +0000 UTC m=+66.133911172 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/348b64e9-4db6-4d74-9a29-36436b02e1d3-service-ca-bundle") pod "router-default-8c8d8cb74-pbqx5" (UID: "348b64e9-4db6-4d74-9a29-36436b02e1d3") : configmap references non-existent config key: service-ca.crt Jul 10 18:59:40.090070 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:40.089800 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 18:59:40.090070 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:40.089848 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/348b64e9-4db6-4d74-9a29-36436b02e1d3-metrics-certs podName:348b64e9-4db6-4d74-9a29-36436b02e1d3 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:42.089833817 +0000 UTC m=+66.133976314 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/348b64e9-4db6-4d74-9a29-36436b02e1d3-metrics-certs") pod "router-default-8c8d8cb74-pbqx5" (UID: "348b64e9-4db6-4d74-9a29-36436b02e1d3") : secret "router-metrics-certs-default" not found Jul 10 18:59:40.930856 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:40.930819 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-68z78" event={"ID":"1f90ac19-a36e-4ee5-ba73-c7f06f7950f9","Type":"ContainerStarted","Data":"9245ec52962a1b53d2b37513bf2c1b4009daa236590ca2429d2ab9d36b9fb432"} Jul 10 18:59:40.948966 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:40.948920 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-68z78" podStartSLOduration=1.74790544 podStartE2EDuration="2.948903545s" totalImagesPullingTime="1.200998105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:59:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:40.948458035 +0000 UTC m=+64.992600563" watchObservedRunningTime="2026-07-10 18:59:40.948903545 +0000 UTC m=+64.993046046" Jul 10 18:59:41.301401 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:41.301359 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwxth\" (UniqueName: \"kubernetes.io/projected/48523b16-8cba-40d8-bbaf-e05814fcb820-kube-api-access-vwxth\") pod \"network-check-target-4fmmt\" (UID: \"48523b16-8cba-40d8-bbaf-e05814fcb820\") " pod="openshift-network-diagnostics/network-check-target-4fmmt" Jul 10 18:59:41.301845 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:41.301447 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71f50128-1d81-4578-8758-cc4b92443331-metrics-certs\") pod \"network-metrics-daemon-brfp2\" (UID: \"71f50128-1d81-4578-8758-cc4b92443331\") " pod="openshift-multus/network-metrics-daemon-brfp2" Jul 10 18:59:41.301845 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:41.301475 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe-original-pull-secret\") pod \"global-pull-secret-syncer-qc4kd\" (UID: \"f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe\") " pod="kube-system/global-pull-secret-syncer-qc4kd" Jul 10 18:59:41.304619 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:41.304594 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 18:59:41.304728 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:41.304704 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 18:59:41.304795 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:41.304711 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 18:59:41.312059 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:41.312039 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 10 18:59:41.312178 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:41.312102 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71f50128-1d81-4578-8758-cc4b92443331-metrics-certs podName:71f50128-1d81-4578-8758-cc4b92443331 nodeName:}" failed. No retries permitted until 2026-07-10 19:00:45.312083498 +0000 UTC m=+129.356225997 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71f50128-1d81-4578-8758-cc4b92443331-metrics-certs") pod "network-metrics-daemon-brfp2" (UID: "71f50128-1d81-4578-8758-cc4b92443331") : secret "metrics-daemon-secret" not found Jul 10 18:59:41.314242 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:41.314213 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe-original-pull-secret\") pod \"global-pull-secret-syncer-qc4kd\" (UID: \"f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe\") " pod="kube-system/global-pull-secret-syncer-qc4kd" Jul 10 18:59:41.314795 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:41.314780 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 18:59:41.325042 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:41.325021 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwxth\" (UniqueName: \"kubernetes.io/projected/48523b16-8cba-40d8-bbaf-e05814fcb820-kube-api-access-vwxth\") pod \"network-check-target-4fmmt\" (UID: \"48523b16-8cba-40d8-bbaf-e05814fcb820\") " pod="openshift-network-diagnostics/network-check-target-4fmmt" Jul 10 18:59:41.550483 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:41.550444 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qc4kd" Jul 10 18:59:41.566604 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:41.566578 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xwgdk\"" Jul 10 18:59:41.574671 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:41.574649 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4fmmt" Jul 10 18:59:41.692403 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:41.692371 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qc4kd"] Jul 10 18:59:41.695907 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:59:41.695879 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7d3d4a3_cb7b_4737_8f95_e3b5aac1b5fe.slice/crio-ff1acc7047f267dd42b53d81b61eae2bbaf96af132d20eb31aaa406e8e10c7c1 WatchSource:0}: Error finding container ff1acc7047f267dd42b53d81b61eae2bbaf96af132d20eb31aaa406e8e10c7c1: Status 404 returned error can't find the container with id ff1acc7047f267dd42b53d81b61eae2bbaf96af132d20eb31aaa406e8e10c7c1 Jul 10 18:59:41.737559 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:41.737527 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4fmmt"] Jul 10 18:59:41.740755 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:59:41.740723 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48523b16_8cba_40d8_bbaf_e05814fcb820.slice/crio-04012900a4e509e0977872f6b102bdc35f210e35315a854343e73bfae2e49b4d WatchSource:0}: Error finding container 04012900a4e509e0977872f6b102bdc35f210e35315a854343e73bfae2e49b4d: Status 404 returned error can't find the container with id 04012900a4e509e0977872f6b102bdc35f210e35315a854343e73bfae2e49b4d Jul 10 18:59:41.934426 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:41.934332 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qc4kd" event={"ID":"f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe","Type":"ContainerStarted","Data":"ff1acc7047f267dd42b53d81b61eae2bbaf96af132d20eb31aaa406e8e10c7c1"} Jul 10 18:59:41.935441 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:41.935411 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4fmmt" event={"ID":"48523b16-8cba-40d8-bbaf-e05814fcb820","Type":"ContainerStarted","Data":"04012900a4e509e0977872f6b102bdc35f210e35315a854343e73bfae2e49b4d"} Jul 10 18:59:42.109131 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:42.109091 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/348b64e9-4db6-4d74-9a29-36436b02e1d3-metrics-certs\") pod \"router-default-8c8d8cb74-pbqx5\" (UID: \"348b64e9-4db6-4d74-9a29-36436b02e1d3\") " pod="openshift-ingress/router-default-8c8d8cb74-pbqx5" Jul 10 18:59:42.109309 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:42.109197 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/348b64e9-4db6-4d74-9a29-36436b02e1d3-service-ca-bundle\") pod \"router-default-8c8d8cb74-pbqx5\" (UID: \"348b64e9-4db6-4d74-9a29-36436b02e1d3\") " pod="openshift-ingress/router-default-8c8d8cb74-pbqx5" Jul 10 18:59:42.109309 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:42.109270 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 18:59:42.109410 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:42.109332 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/348b64e9-4db6-4d74-9a29-36436b02e1d3-service-ca-bundle podName:348b64e9-4db6-4d74-9a29-36436b02e1d3 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:46.109313146 +0000 UTC m=+70.153455626 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/348b64e9-4db6-4d74-9a29-36436b02e1d3-service-ca-bundle") pod "router-default-8c8d8cb74-pbqx5" (UID: "348b64e9-4db6-4d74-9a29-36436b02e1d3") : configmap references non-existent config key: service-ca.crt Jul 10 18:59:42.109410 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:42.109354 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/348b64e9-4db6-4d74-9a29-36436b02e1d3-metrics-certs podName:348b64e9-4db6-4d74-9a29-36436b02e1d3 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:46.109345128 +0000 UTC m=+70.153487614 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/348b64e9-4db6-4d74-9a29-36436b02e1d3-metrics-certs") pod "router-default-8c8d8cb74-pbqx5" (UID: "348b64e9-4db6-4d74-9a29-36436b02e1d3") : secret "router-metrics-certs-default" not found Jul 10 18:59:44.128782 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:44.128742 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5fqvm"] Jul 10 18:59:44.131284 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:44.131258 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5fqvm" Jul 10 18:59:44.134795 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:44.134668 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 10 18:59:44.134795 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:44.134679 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 10 18:59:44.135189 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:44.135170 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 10 18:59:44.135305 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:44.135287 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 10 18:59:44.136548 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:44.136527 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8f6gd\"" Jul 10 18:59:44.145925 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:44.145902 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5fqvm"] Jul 10 18:59:44.224319 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:44.224281 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/89e10ee7-fb7a-4cc2-9638-030cf5c0a305-data-volume\") pod \"insights-runtime-extractor-5fqvm\" (UID: \"89e10ee7-fb7a-4cc2-9638-030cf5c0a305\") " pod="openshift-insights/insights-runtime-extractor-5fqvm" Jul 10 18:59:44.224481 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:44.224331 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk2jp\" (UniqueName: \"kubernetes.io/projected/89e10ee7-fb7a-4cc2-9638-030cf5c0a305-kube-api-access-mk2jp\") pod \"insights-runtime-extractor-5fqvm\" (UID: \"89e10ee7-fb7a-4cc2-9638-030cf5c0a305\") " pod="openshift-insights/insights-runtime-extractor-5fqvm" Jul 10 18:59:44.224481 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:44.224410 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/89e10ee7-fb7a-4cc2-9638-030cf5c0a305-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5fqvm\" (UID: \"89e10ee7-fb7a-4cc2-9638-030cf5c0a305\") " pod="openshift-insights/insights-runtime-extractor-5fqvm" Jul 10 18:59:44.224481 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:44.224437 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89e10ee7-fb7a-4cc2-9638-030cf5c0a305-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5fqvm\" (UID: \"89e10ee7-fb7a-4cc2-9638-030cf5c0a305\") " pod="openshift-insights/insights-runtime-extractor-5fqvm" Jul 10 18:59:44.224481 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:44.224467 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/89e10ee7-fb7a-4cc2-9638-030cf5c0a305-crio-socket\") pod \"insights-runtime-extractor-5fqvm\" (UID: \"89e10ee7-fb7a-4cc2-9638-030cf5c0a305\") " pod="openshift-insights/insights-runtime-extractor-5fqvm" Jul 10 18:59:44.325845 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:44.325809 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/89e10ee7-fb7a-4cc2-9638-030cf5c0a305-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5fqvm\" (UID: \"89e10ee7-fb7a-4cc2-9638-030cf5c0a305\") " pod="openshift-insights/insights-runtime-extractor-5fqvm" Jul 10 18:59:44.326026 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:44.325850 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89e10ee7-fb7a-4cc2-9638-030cf5c0a305-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5fqvm\" (UID: \"89e10ee7-fb7a-4cc2-9638-030cf5c0a305\") " pod="openshift-insights/insights-runtime-extractor-5fqvm" Jul 10 18:59:44.326026 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:44.325877 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/89e10ee7-fb7a-4cc2-9638-030cf5c0a305-crio-socket\") pod \"insights-runtime-extractor-5fqvm\" (UID: \"89e10ee7-fb7a-4cc2-9638-030cf5c0a305\") " pod="openshift-insights/insights-runtime-extractor-5fqvm" Jul 10 18:59:44.326026 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:44.325938 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/89e10ee7-fb7a-4cc2-9638-030cf5c0a305-data-volume\") pod \"insights-runtime-extractor-5fqvm\" (UID: \"89e10ee7-fb7a-4cc2-9638-030cf5c0a305\") " pod="openshift-insights/insights-runtime-extractor-5fqvm" Jul 10 18:59:44.326026 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:44.325949 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 18:59:44.326026 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:44.325962 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk2jp\" (UniqueName: \"kubernetes.io/projected/89e10ee7-fb7a-4cc2-9638-030cf5c0a305-kube-api-access-mk2jp\") pod \"insights-runtime-extractor-5fqvm\" (UID: \"89e10ee7-fb7a-4cc2-9638-030cf5c0a305\") " pod="openshift-insights/insights-runtime-extractor-5fqvm" Jul 10 18:59:44.326227 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:44.326030 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/89e10ee7-fb7a-4cc2-9638-030cf5c0a305-kube-rbac-proxy-cm podName:89e10ee7-fb7a-4cc2-9638-030cf5c0a305 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:44.826007818 +0000 UTC m=+68.870150313 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/89e10ee7-fb7a-4cc2-9638-030cf5c0a305-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5fqvm" (UID: "89e10ee7-fb7a-4cc2-9638-030cf5c0a305") : configmap "kube-rbac-proxy" not found Jul 10 18:59:44.326227 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:44.326029 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 18:59:44.326227 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:44.326069 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89e10ee7-fb7a-4cc2-9638-030cf5c0a305-insights-runtime-extractor-tls podName:89e10ee7-fb7a-4cc2-9638-030cf5c0a305 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:44.826059146 +0000 UTC m=+68.870201632 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/89e10ee7-fb7a-4cc2-9638-030cf5c0a305-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5fqvm" (UID: "89e10ee7-fb7a-4cc2-9638-030cf5c0a305") : secret "insights-runtime-extractor-tls" not found Jul 10 18:59:44.326227 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:44.326091 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/89e10ee7-fb7a-4cc2-9638-030cf5c0a305-crio-socket\") pod \"insights-runtime-extractor-5fqvm\" (UID: \"89e10ee7-fb7a-4cc2-9638-030cf5c0a305\") " pod="openshift-insights/insights-runtime-extractor-5fqvm" Jul 10 18:59:44.326451 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:44.326434 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/89e10ee7-fb7a-4cc2-9638-030cf5c0a305-data-volume\") pod \"insights-runtime-extractor-5fqvm\" (UID: \"89e10ee7-fb7a-4cc2-9638-030cf5c0a305\") " pod="openshift-insights/insights-runtime-extractor-5fqvm" Jul 10 18:59:44.334852 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:44.334827 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk2jp\" (UniqueName: \"kubernetes.io/projected/89e10ee7-fb7a-4cc2-9638-030cf5c0a305-kube-api-access-mk2jp\") pod \"insights-runtime-extractor-5fqvm\" (UID: \"89e10ee7-fb7a-4cc2-9638-030cf5c0a305\") " pod="openshift-insights/insights-runtime-extractor-5fqvm" Jul 10 18:59:44.649022 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:44.648992 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-dkp8t"] Jul 10 18:59:44.653362 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:44.653338 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-dkp8t" Jul 10 18:59:44.655916 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:44.655885 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 10 18:59:44.656022 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:44.655883 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 10 18:59:44.656195 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:44.656182 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-fdsff\"" Jul 10 18:59:44.663842 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:44.663818 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-dkp8t"] Jul 10 18:59:44.730840 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:44.730809 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ed9ac220-276d-4864-b19b-17b8f69eaa98-metrics-tls\") pod \"dns-default-v5rql\" (UID: \"ed9ac220-276d-4864-b19b-17b8f69eaa98\") " pod="openshift-dns/dns-default-v5rql" Jul 10 18:59:44.731042 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:44.730872 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm5z4\" (UniqueName: \"kubernetes.io/projected/e7b8e06d-2dd5-4468-b6f8-4d62c2f6aac0-kube-api-access-vm5z4\") pod \"migrator-7f59b54f6c-dkp8t\" (UID: \"e7b8e06d-2dd5-4468-b6f8-4d62c2f6aac0\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-dkp8t" Jul 10 18:59:44.731042 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:44.731022 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 18:59:44.731155 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:44.731099 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed9ac220-276d-4864-b19b-17b8f69eaa98-metrics-tls podName:ed9ac220-276d-4864-b19b-17b8f69eaa98 nodeName:}" failed. No retries permitted until 2026-07-10 19:00:16.731077136 +0000 UTC m=+100.775219632 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ed9ac220-276d-4864-b19b-17b8f69eaa98-metrics-tls") pod "dns-default-v5rql" (UID: "ed9ac220-276d-4864-b19b-17b8f69eaa98") : secret "dns-default-metrics-tls" not found Jul 10 18:59:44.831604 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:44.831565 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/89e10ee7-fb7a-4cc2-9638-030cf5c0a305-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5fqvm\" (UID: \"89e10ee7-fb7a-4cc2-9638-030cf5c0a305\") " pod="openshift-insights/insights-runtime-extractor-5fqvm" Jul 10 18:59:44.831789 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:44.831616 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89e10ee7-fb7a-4cc2-9638-030cf5c0a305-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5fqvm\" (UID: \"89e10ee7-fb7a-4cc2-9638-030cf5c0a305\") " pod="openshift-insights/insights-runtime-extractor-5fqvm" Jul 10 18:59:44.831789 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:44.831645 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm5z4\" (UniqueName: \"kubernetes.io/projected/e7b8e06d-2dd5-4468-b6f8-4d62c2f6aac0-kube-api-access-vm5z4\") pod \"migrator-7f59b54f6c-dkp8t\" (UID: \"e7b8e06d-2dd5-4468-b6f8-4d62c2f6aac0\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-dkp8t" Jul 10 18:59:44.831789 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:44.831666 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 18:59:44.831789 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:44.831768 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/89e10ee7-fb7a-4cc2-9638-030cf5c0a305-kube-rbac-proxy-cm podName:89e10ee7-fb7a-4cc2-9638-030cf5c0a305 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:45.831745257 +0000 UTC m=+69.875887738 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/89e10ee7-fb7a-4cc2-9638-030cf5c0a305-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5fqvm" (UID: "89e10ee7-fb7a-4cc2-9638-030cf5c0a305") : configmap "kube-rbac-proxy" not found Jul 10 18:59:44.831999 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:44.831799 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 18:59:44.831999 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:44.831858 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89e10ee7-fb7a-4cc2-9638-030cf5c0a305-insights-runtime-extractor-tls podName:89e10ee7-fb7a-4cc2-9638-030cf5c0a305 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:45.831841488 +0000 UTC m=+69.875983971 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/89e10ee7-fb7a-4cc2-9638-030cf5c0a305-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5fqvm" (UID: "89e10ee7-fb7a-4cc2-9638-030cf5c0a305") : secret "insights-runtime-extractor-tls" not found Jul 10 18:59:44.843042 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:44.843017 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm5z4\" (UniqueName: \"kubernetes.io/projected/e7b8e06d-2dd5-4468-b6f8-4d62c2f6aac0-kube-api-access-vm5z4\") pod \"migrator-7f59b54f6c-dkp8t\" (UID: \"e7b8e06d-2dd5-4468-b6f8-4d62c2f6aac0\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-dkp8t" Jul 10 18:59:44.962581 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:44.962498 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-dkp8t" Jul 10 18:59:45.839916 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:45.839886 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/89e10ee7-fb7a-4cc2-9638-030cf5c0a305-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5fqvm\" (UID: \"89e10ee7-fb7a-4cc2-9638-030cf5c0a305\") " pod="openshift-insights/insights-runtime-extractor-5fqvm" Jul 10 18:59:45.839916 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:45.839919 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89e10ee7-fb7a-4cc2-9638-030cf5c0a305-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5fqvm\" (UID: \"89e10ee7-fb7a-4cc2-9638-030cf5c0a305\") " pod="openshift-insights/insights-runtime-extractor-5fqvm" Jul 10 18:59:45.840400 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:45.840026 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 18:59:45.840400 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:45.840073 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 18:59:45.840400 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:45.840121 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/89e10ee7-fb7a-4cc2-9638-030cf5c0a305-kube-rbac-proxy-cm podName:89e10ee7-fb7a-4cc2-9638-030cf5c0a305 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:47.840099378 +0000 UTC m=+71.884241874 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/89e10ee7-fb7a-4cc2-9638-030cf5c0a305-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5fqvm" (UID: "89e10ee7-fb7a-4cc2-9638-030cf5c0a305") : configmap "kube-rbac-proxy" not found Jul 10 18:59:45.840400 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:45.840142 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89e10ee7-fb7a-4cc2-9638-030cf5c0a305-insights-runtime-extractor-tls podName:89e10ee7-fb7a-4cc2-9638-030cf5c0a305 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:47.840131775 +0000 UTC m=+71.884274260 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/89e10ee7-fb7a-4cc2-9638-030cf5c0a305-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5fqvm" (UID: "89e10ee7-fb7a-4cc2-9638-030cf5c0a305") : secret "insights-runtime-extractor-tls" not found Jul 10 18:59:46.142160 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:46.142070 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/348b64e9-4db6-4d74-9a29-36436b02e1d3-service-ca-bundle\") pod \"router-default-8c8d8cb74-pbqx5\" (UID: \"348b64e9-4db6-4d74-9a29-36436b02e1d3\") " pod="openshift-ingress/router-default-8c8d8cb74-pbqx5" Jul 10 18:59:46.142160 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:46.142129 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/348b64e9-4db6-4d74-9a29-36436b02e1d3-metrics-certs\") pod \"router-default-8c8d8cb74-pbqx5\" (UID: \"348b64e9-4db6-4d74-9a29-36436b02e1d3\") " pod="openshift-ingress/router-default-8c8d8cb74-pbqx5" Jul 10 18:59:46.142362 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:46.142238 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/348b64e9-4db6-4d74-9a29-36436b02e1d3-service-ca-bundle podName:348b64e9-4db6-4d74-9a29-36436b02e1d3 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:54.142219718 +0000 UTC m=+78.186362198 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/348b64e9-4db6-4d74-9a29-36436b02e1d3-service-ca-bundle") pod "router-default-8c8d8cb74-pbqx5" (UID: "348b64e9-4db6-4d74-9a29-36436b02e1d3") : configmap references non-existent config key: service-ca.crt Jul 10 18:59:46.142362 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:46.142298 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 18:59:46.142362 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:46.142352 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/348b64e9-4db6-4d74-9a29-36436b02e1d3-metrics-certs podName:348b64e9-4db6-4d74-9a29-36436b02e1d3 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:54.142340587 +0000 UTC m=+78.186483067 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/348b64e9-4db6-4d74-9a29-36436b02e1d3-metrics-certs") pod "router-default-8c8d8cb74-pbqx5" (UID: "348b64e9-4db6-4d74-9a29-36436b02e1d3") : secret "router-metrics-certs-default" not found Jul 10 18:59:46.301285 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:46.301256 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zf82d_df38c8b2-26d6-46bb-95ca-166ff4723ceb/dns-node-resolver/0.log" Jul 10 18:59:46.601802 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:46.601773 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-dkp8t"] Jul 10 18:59:46.605067 ip-10-0-135-67 kubenswrapper[2581]: W0710 18:59:46.605042 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7b8e06d_2dd5_4468_b6f8_4d62c2f6aac0.slice/crio-cd913061937467b140a763e5931e88a4bf4bb658627790190323d89236e73de8 WatchSource:0}: Error finding container cd913061937467b140a763e5931e88a4bf4bb658627790190323d89236e73de8: Status 404 returned error can't find the container with id cd913061937467b140a763e5931e88a4bf4bb658627790190323d89236e73de8 Jul 10 18:59:46.904664 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:46.904640 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7btwf_10a5a151-ab9d-45db-8502-26d00ec21659/node-ca/0.log" Jul 10 18:59:46.950890 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:46.950855 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qc4kd" event={"ID":"f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe","Type":"ContainerStarted","Data":"2f04ed93dada8b05d8fb3b8b70d499343d5394bea34ca775d93599aba5f32ad1"} Jul 10 18:59:46.951968 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:46.951936 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-dkp8t" event={"ID":"e7b8e06d-2dd5-4468-b6f8-4d62c2f6aac0","Type":"ContainerStarted","Data":"cd913061937467b140a763e5931e88a4bf4bb658627790190323d89236e73de8"} Jul 10 18:59:46.953283 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:46.953262 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4fmmt" event={"ID":"48523b16-8cba-40d8-bbaf-e05814fcb820","Type":"ContainerStarted","Data":"3a2385e9a306e47780ac1f3db7d1a9d595e4958dee5222cbf7af770f0e2994bd"} Jul 10 18:59:46.953397 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:46.953384 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4fmmt" Jul 10 18:59:46.977415 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:46.977367 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-qc4kd" podStartSLOduration=65.194764305 podStartE2EDuration="1m9.977349859s" totalImagesPullingTime="4.782585554s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:46.968256457 +0000 UTC m=+71.012398955" watchObservedRunningTime="2026-07-10 18:59:46.977349859 +0000 UTC m=+71.021492370" Jul 10 18:59:46.999276 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:46.999229 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-4fmmt" podStartSLOduration=65.272849876 podStartE2EDuration="1m9.999210832s" totalImagesPullingTime="4.726360956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:46.997849054 +0000 UTC m=+71.041991577" watchObservedRunningTime="2026-07-10 18:59:46.999210832 +0000 UTC m=+71.043353333" Jul 10 18:59:47.855484 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:47.855452 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/89e10ee7-fb7a-4cc2-9638-030cf5c0a305-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5fqvm\" (UID: \"89e10ee7-fb7a-4cc2-9638-030cf5c0a305\") " pod="openshift-insights/insights-runtime-extractor-5fqvm" Jul 10 18:59:47.855484 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:47.855487 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89e10ee7-fb7a-4cc2-9638-030cf5c0a305-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5fqvm\" (UID: \"89e10ee7-fb7a-4cc2-9638-030cf5c0a305\") " pod="openshift-insights/insights-runtime-extractor-5fqvm" Jul 10 18:59:47.855722 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:47.855551 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 18:59:47.855722 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:47.855589 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 18:59:47.855722 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:47.855633 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/89e10ee7-fb7a-4cc2-9638-030cf5c0a305-kube-rbac-proxy-cm podName:89e10ee7-fb7a-4cc2-9638-030cf5c0a305 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:51.855611358 +0000 UTC m=+75.899753852 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/89e10ee7-fb7a-4cc2-9638-030cf5c0a305-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5fqvm" (UID: "89e10ee7-fb7a-4cc2-9638-030cf5c0a305") : configmap "kube-rbac-proxy" not found Jul 10 18:59:47.855722 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:47.855652 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89e10ee7-fb7a-4cc2-9638-030cf5c0a305-insights-runtime-extractor-tls podName:89e10ee7-fb7a-4cc2-9638-030cf5c0a305 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:51.855643221 +0000 UTC m=+75.899785731 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/89e10ee7-fb7a-4cc2-9638-030cf5c0a305-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5fqvm" (UID: "89e10ee7-fb7a-4cc2-9638-030cf5c0a305") : secret "insights-runtime-extractor-tls" not found Jul 10 18:59:47.906710 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:47.906673 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-szfx9_3a2ad480-1b70-460f-8fe5-e0dafed541e5/serve-healthcheck-canary/0.log" Jul 10 18:59:47.957671 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:47.957647 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-dkp8t" event={"ID":"e7b8e06d-2dd5-4468-b6f8-4d62c2f6aac0","Type":"ContainerStarted","Data":"898930f051e4e21c7fee1dcc7aa113282834064b98f979b6aa03095384f0f480"} Jul 10 18:59:48.963480 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:48.963442 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-dkp8t" event={"ID":"e7b8e06d-2dd5-4468-b6f8-4d62c2f6aac0","Type":"ContainerStarted","Data":"733457c559252525eafe640c3f6d74d2c6980ae73c472026db07c1f1bbbcc471"} Jul 10 18:59:48.983373 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:48.983325 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-dkp8t" podStartSLOduration=3.732007617 podStartE2EDuration="4.983287112s" totalImagesPullingTime="1.251279495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:59:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:48.983059163 +0000 UTC m=+73.027201663" watchObservedRunningTime="2026-07-10 18:59:48.983287112 +0000 UTC m=+73.027429613" Jul 10 18:59:51.883643 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:51.883587 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/89e10ee7-fb7a-4cc2-9638-030cf5c0a305-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5fqvm\" (UID: \"89e10ee7-fb7a-4cc2-9638-030cf5c0a305\") " pod="openshift-insights/insights-runtime-extractor-5fqvm" Jul 10 18:59:51.883643 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:51.883645 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89e10ee7-fb7a-4cc2-9638-030cf5c0a305-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5fqvm\" (UID: \"89e10ee7-fb7a-4cc2-9638-030cf5c0a305\") " pod="openshift-insights/insights-runtime-extractor-5fqvm" Jul 10 18:59:51.884195 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:51.883723 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 18:59:51.884195 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:51.883788 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/89e10ee7-fb7a-4cc2-9638-030cf5c0a305-kube-rbac-proxy-cm podName:89e10ee7-fb7a-4cc2-9638-030cf5c0a305 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:59.883773322 +0000 UTC m=+83.927915801 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/89e10ee7-fb7a-4cc2-9638-030cf5c0a305-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5fqvm" (UID: "89e10ee7-fb7a-4cc2-9638-030cf5c0a305") : configmap "kube-rbac-proxy" not found Jul 10 18:59:51.884195 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:51.883809 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 18:59:51.884195 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:51.883857 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89e10ee7-fb7a-4cc2-9638-030cf5c0a305-insights-runtime-extractor-tls podName:89e10ee7-fb7a-4cc2-9638-030cf5c0a305 nodeName:}" failed. No retries permitted until 2026-07-10 18:59:59.883844414 +0000 UTC m=+83.927986893 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/89e10ee7-fb7a-4cc2-9638-030cf5c0a305-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5fqvm" (UID: "89e10ee7-fb7a-4cc2-9638-030cf5c0a305") : secret "insights-runtime-extractor-tls" not found Jul 10 18:59:54.203633 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:54.203583 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/348b64e9-4db6-4d74-9a29-36436b02e1d3-service-ca-bundle\") pod \"router-default-8c8d8cb74-pbqx5\" (UID: \"348b64e9-4db6-4d74-9a29-36436b02e1d3\") " pod="openshift-ingress/router-default-8c8d8cb74-pbqx5" Jul 10 18:59:54.203633 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:54.203645 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/348b64e9-4db6-4d74-9a29-36436b02e1d3-metrics-certs\") pod \"router-default-8c8d8cb74-pbqx5\" (UID: \"348b64e9-4db6-4d74-9a29-36436b02e1d3\") " pod="openshift-ingress/router-default-8c8d8cb74-pbqx5" Jul 10 18:59:54.204117 ip-10-0-135-67 kubenswrapper[2581]: E0710 18:59:54.203763 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/348b64e9-4db6-4d74-9a29-36436b02e1d3-service-ca-bundle podName:348b64e9-4db6-4d74-9a29-36436b02e1d3 nodeName:}" failed. No retries permitted until 2026-07-10 19:00:10.203745755 +0000 UTC m=+94.247888235 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/348b64e9-4db6-4d74-9a29-36436b02e1d3-service-ca-bundle") pod "router-default-8c8d8cb74-pbqx5" (UID: "348b64e9-4db6-4d74-9a29-36436b02e1d3") : configmap references non-existent config key: service-ca.crt Jul 10 18:59:54.205992 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:54.205969 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/348b64e9-4db6-4d74-9a29-36436b02e1d3-metrics-certs\") pod \"router-default-8c8d8cb74-pbqx5\" (UID: \"348b64e9-4db6-4d74-9a29-36436b02e1d3\") " pod="openshift-ingress/router-default-8c8d8cb74-pbqx5" Jul 10 18:59:59.946072 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:59.945943 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/89e10ee7-fb7a-4cc2-9638-030cf5c0a305-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5fqvm\" (UID: \"89e10ee7-fb7a-4cc2-9638-030cf5c0a305\") " pod="openshift-insights/insights-runtime-extractor-5fqvm" Jul 10 18:59:59.946072 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:59.945985 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89e10ee7-fb7a-4cc2-9638-030cf5c0a305-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5fqvm\" (UID: \"89e10ee7-fb7a-4cc2-9638-030cf5c0a305\") " pod="openshift-insights/insights-runtime-extractor-5fqvm" Jul 10 18:59:59.946509 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:59.946485 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/89e10ee7-fb7a-4cc2-9638-030cf5c0a305-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5fqvm\" (UID: \"89e10ee7-fb7a-4cc2-9638-030cf5c0a305\") " pod="openshift-insights/insights-runtime-extractor-5fqvm" Jul 10 18:59:59.948272 ip-10-0-135-67 kubenswrapper[2581]: I0710 18:59:59.948250 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89e10ee7-fb7a-4cc2-9638-030cf5c0a305-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5fqvm\" (UID: \"89e10ee7-fb7a-4cc2-9638-030cf5c0a305\") " pod="openshift-insights/insights-runtime-extractor-5fqvm" Jul 10 19:00:00.043265 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:00.043223 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5fqvm" Jul 10 19:00:00.171042 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:00.171007 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5fqvm"] Jul 10 19:00:00.174161 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:00:00.174138 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89e10ee7_fb7a_4cc2_9638_030cf5c0a305.slice/crio-4564ced861b7fd6ea6775a2a0f3a8b03c1f7ebec006347235d73d20d3fd045a7 WatchSource:0}: Error finding container 4564ced861b7fd6ea6775a2a0f3a8b03c1f7ebec006347235d73d20d3fd045a7: Status 404 returned error can't find the container with id 4564ced861b7fd6ea6775a2a0f3a8b03c1f7ebec006347235d73d20d3fd045a7 Jul 10 19:00:01.000882 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:01.000844 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5fqvm" event={"ID":"89e10ee7-fb7a-4cc2-9638-030cf5c0a305","Type":"ContainerStarted","Data":"4564ced861b7fd6ea6775a2a0f3a8b03c1f7ebec006347235d73d20d3fd045a7"} Jul 10 19:00:04.010990 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:04.010957 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5fqvm" event={"ID":"89e10ee7-fb7a-4cc2-9638-030cf5c0a305","Type":"ContainerStarted","Data":"f8557a67b3e967f99a52164ca55454685ab54cc86e2422ed84f8afc47cc8afd0"} Jul 10 19:00:05.015079 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:05.015042 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5fqvm" event={"ID":"89e10ee7-fb7a-4cc2-9638-030cf5c0a305","Type":"ContainerStarted","Data":"f48dbb6730344db614ed2221dbafc2729ff604c2bc07956437dfc64b1e6a9c01"} Jul 10 19:00:06.324595 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:06.324555 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-66d5548f5b-bvl7g"] Jul 10 19:00:06.328724 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:06.328698 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66d5548f5b-bvl7g" Jul 10 19:00:06.331433 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:06.331408 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 10 19:00:06.332440 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:06.332421 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 10 19:00:06.332643 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:06.332628 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 10 19:00:06.340775 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:06.340753 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 10 19:00:06.348270 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:06.348238 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-66d5548f5b-bvl7g"] Jul 10 19:00:06.395452 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:06.395419 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca9cf55a-1bb3-4515-857a-3bd7f9c11795-tmp\") pod \"klusterlet-addon-workmgr-66d5548f5b-bvl7g\" (UID: \"ca9cf55a-1bb3-4515-857a-3bd7f9c11795\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66d5548f5b-bvl7g" Jul 10 19:00:06.395639 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:06.395462 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vxf8\" (UniqueName: \"kubernetes.io/projected/ca9cf55a-1bb3-4515-857a-3bd7f9c11795-kube-api-access-9vxf8\") pod \"klusterlet-addon-workmgr-66d5548f5b-bvl7g\" (UID: \"ca9cf55a-1bb3-4515-857a-3bd7f9c11795\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66d5548f5b-bvl7g" Jul 10 19:00:06.395639 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:06.395545 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ca9cf55a-1bb3-4515-857a-3bd7f9c11795-klusterlet-config\") pod \"klusterlet-addon-workmgr-66d5548f5b-bvl7g\" (UID: \"ca9cf55a-1bb3-4515-857a-3bd7f9c11795\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66d5548f5b-bvl7g" Jul 10 19:00:06.406835 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:06.406803 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-65859cd4cb-2n5rm"] Jul 10 19:00:06.409987 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:06.409963 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65859cd4cb-2n5rm" Jul 10 19:00:06.412713 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:06.412564 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 10 19:00:06.412713 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:06.412642 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-d494m\"" Jul 10 19:00:06.412713 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:06.412711 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 10 19:00:06.412942 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:06.412752 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 10 19:00:06.418328 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:06.418307 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 10 19:00:06.426998 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:06.426974 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65859cd4cb-2n5rm"] Jul 10 19:00:06.496172 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:06.496133 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f8c1fa0c-a239-48b9-8dae-f94be920ae1a-image-registry-private-configuration\") pod \"image-registry-65859cd4cb-2n5rm\" (UID: \"f8c1fa0c-a239-48b9-8dae-f94be920ae1a\") " pod="openshift-image-registry/image-registry-65859cd4cb-2n5rm" Jul 10 19:00:06.496369 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:06.496188 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca9cf55a-1bb3-4515-857a-3bd7f9c11795-tmp\") pod \"klusterlet-addon-workmgr-66d5548f5b-bvl7g\" (UID: \"ca9cf55a-1bb3-4515-857a-3bd7f9c11795\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66d5548f5b-bvl7g" Jul 10 19:00:06.496369 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:06.496214 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f8c1fa0c-a239-48b9-8dae-f94be920ae1a-registry-tls\") pod \"image-registry-65859cd4cb-2n5rm\" (UID: \"f8c1fa0c-a239-48b9-8dae-f94be920ae1a\") " pod="openshift-image-registry/image-registry-65859cd4cb-2n5rm" Jul 10 19:00:06.496369 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:06.496258 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vxf8\" (UniqueName: \"kubernetes.io/projected/ca9cf55a-1bb3-4515-857a-3bd7f9c11795-kube-api-access-9vxf8\") pod \"klusterlet-addon-workmgr-66d5548f5b-bvl7g\" (UID: \"ca9cf55a-1bb3-4515-857a-3bd7f9c11795\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66d5548f5b-bvl7g" Jul 10 19:00:06.496369 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:06.496283 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f8c1fa0c-a239-48b9-8dae-f94be920ae1a-registry-certificates\") pod \"image-registry-65859cd4cb-2n5rm\" (UID: \"f8c1fa0c-a239-48b9-8dae-f94be920ae1a\") " pod="openshift-image-registry/image-registry-65859cd4cb-2n5rm" Jul 10 19:00:06.496369 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:06.496310 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ca9cf55a-1bb3-4515-857a-3bd7f9c11795-klusterlet-config\") pod \"klusterlet-addon-workmgr-66d5548f5b-bvl7g\" (UID: \"ca9cf55a-1bb3-4515-857a-3bd7f9c11795\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66d5548f5b-bvl7g" Jul 10 19:00:06.496621 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:06.496449 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f8c1fa0c-a239-48b9-8dae-f94be920ae1a-ca-trust-extracted\") pod \"image-registry-65859cd4cb-2n5rm\" (UID: \"f8c1fa0c-a239-48b9-8dae-f94be920ae1a\") " pod="openshift-image-registry/image-registry-65859cd4cb-2n5rm" Jul 10 19:00:06.496621 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:06.496507 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f8c1fa0c-a239-48b9-8dae-f94be920ae1a-bound-sa-token\") pod \"image-registry-65859cd4cb-2n5rm\" (UID: \"f8c1fa0c-a239-48b9-8dae-f94be920ae1a\") " pod="openshift-image-registry/image-registry-65859cd4cb-2n5rm" Jul 10 19:00:06.496621 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:06.496548 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f8c1fa0c-a239-48b9-8dae-f94be920ae1a-trusted-ca\") pod \"image-registry-65859cd4cb-2n5rm\" (UID: \"f8c1fa0c-a239-48b9-8dae-f94be920ae1a\") " pod="openshift-image-registry/image-registry-65859cd4cb-2n5rm" Jul 10 19:00:06.496621 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:06.496573 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f8c1fa0c-a239-48b9-8dae-f94be920ae1a-installation-pull-secrets\") pod \"image-registry-65859cd4cb-2n5rm\" (UID: \"f8c1fa0c-a239-48b9-8dae-f94be920ae1a\") " pod="openshift-image-registry/image-registry-65859cd4cb-2n5rm" Jul 10 19:00:06.496621 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:06.496596 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl2pp\" (UniqueName: \"kubernetes.io/projected/f8c1fa0c-a239-48b9-8dae-f94be920ae1a-kube-api-access-bl2pp\") pod \"image-registry-65859cd4cb-2n5rm\" (UID: \"f8c1fa0c-a239-48b9-8dae-f94be920ae1a\") " pod="openshift-image-registry/image-registry-65859cd4cb-2n5rm" Jul 10 19:00:06.496621 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:06.496617 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca9cf55a-1bb3-4515-857a-3bd7f9c11795-tmp\") pod \"klusterlet-addon-workmgr-66d5548f5b-bvl7g\" (UID: \"ca9cf55a-1bb3-4515-857a-3bd7f9c11795\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66d5548f5b-bvl7g" Jul 10 19:00:06.499465 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:06.499436 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ca9cf55a-1bb3-4515-857a-3bd7f9c11795-klusterlet-config\") pod \"klusterlet-addon-workmgr-66d5548f5b-bvl7g\" (UID: \"ca9cf55a-1bb3-4515-857a-3bd7f9c11795\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66d5548f5b-bvl7g" Jul 10 19:00:06.506375 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:06.506192 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vxf8\" (UniqueName: \"kubernetes.io/projected/ca9cf55a-1bb3-4515-857a-3bd7f9c11795-kube-api-access-9vxf8\") pod \"klusterlet-addon-workmgr-66d5548f5b-bvl7g\" (UID: \"ca9cf55a-1bb3-4515-857a-3bd7f9c11795\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66d5548f5b-bvl7g" Jul 10 19:00:06.597942 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:06.597856 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f8c1fa0c-a239-48b9-8dae-f94be920ae1a-image-registry-private-configuration\") pod \"image-registry-65859cd4cb-2n5rm\" (UID: \"f8c1fa0c-a239-48b9-8dae-f94be920ae1a\") " pod="openshift-image-registry/image-registry-65859cd4cb-2n5rm" Jul 10 19:00:06.597942 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:06.597904 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f8c1fa0c-a239-48b9-8dae-f94be920ae1a-registry-tls\") pod \"image-registry-65859cd4cb-2n5rm\" (UID: \"f8c1fa0c-a239-48b9-8dae-f94be920ae1a\") " pod="openshift-image-registry/image-registry-65859cd4cb-2n5rm" Jul 10 19:00:06.597942 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:06.597933 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f8c1fa0c-a239-48b9-8dae-f94be920ae1a-registry-certificates\") pod \"image-registry-65859cd4cb-2n5rm\" (UID: \"f8c1fa0c-a239-48b9-8dae-f94be920ae1a\") " pod="openshift-image-registry/image-registry-65859cd4cb-2n5rm" Jul 10 19:00:06.598208 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:06.597961 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f8c1fa0c-a239-48b9-8dae-f94be920ae1a-ca-trust-extracted\") pod \"image-registry-65859cd4cb-2n5rm\" (UID: \"f8c1fa0c-a239-48b9-8dae-f94be920ae1a\") " pod="openshift-image-registry/image-registry-65859cd4cb-2n5rm" Jul 10 19:00:06.598208 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:06.597988 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f8c1fa0c-a239-48b9-8dae-f94be920ae1a-bound-sa-token\") pod \"image-registry-65859cd4cb-2n5rm\" (UID: \"f8c1fa0c-a239-48b9-8dae-f94be920ae1a\") " pod="openshift-image-registry/image-registry-65859cd4cb-2n5rm" Jul 10 19:00:06.598208 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:06.598019 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f8c1fa0c-a239-48b9-8dae-f94be920ae1a-trusted-ca\") pod \"image-registry-65859cd4cb-2n5rm\" (UID: \"f8c1fa0c-a239-48b9-8dae-f94be920ae1a\") " pod="openshift-image-registry/image-registry-65859cd4cb-2n5rm" Jul 10 19:00:06.598208 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:06.598042 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f8c1fa0c-a239-48b9-8dae-f94be920ae1a-installation-pull-secrets\") pod \"image-registry-65859cd4cb-2n5rm\" (UID: \"f8c1fa0c-a239-48b9-8dae-f94be920ae1a\") " pod="openshift-image-registry/image-registry-65859cd4cb-2n5rm" Jul 10 19:00:06.598208 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:06.598071 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl2pp\" (UniqueName: \"kubernetes.io/projected/f8c1fa0c-a239-48b9-8dae-f94be920ae1a-kube-api-access-bl2pp\") pod \"image-registry-65859cd4cb-2n5rm\" (UID: \"f8c1fa0c-a239-48b9-8dae-f94be920ae1a\") " pod="openshift-image-registry/image-registry-65859cd4cb-2n5rm" Jul 10 19:00:06.599730 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:06.599617 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f8c1fa0c-a239-48b9-8dae-f94be920ae1a-registry-certificates\") pod \"image-registry-65859cd4cb-2n5rm\" (UID: \"f8c1fa0c-a239-48b9-8dae-f94be920ae1a\") " pod="openshift-image-registry/image-registry-65859cd4cb-2n5rm" Jul 10 19:00:06.599730 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:06.599648 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f8c1fa0c-a239-48b9-8dae-f94be920ae1a-ca-trust-extracted\") pod \"image-registry-65859cd4cb-2n5rm\" (UID: \"f8c1fa0c-a239-48b9-8dae-f94be920ae1a\") " pod="openshift-image-registry/image-registry-65859cd4cb-2n5rm" Jul 10 19:00:06.599730 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:06.599621 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f8c1fa0c-a239-48b9-8dae-f94be920ae1a-trusted-ca\") pod \"image-registry-65859cd4cb-2n5rm\" (UID: \"f8c1fa0c-a239-48b9-8dae-f94be920ae1a\") " pod="openshift-image-registry/image-registry-65859cd4cb-2n5rm" Jul 10 19:00:06.604506 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:06.601132 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f8c1fa0c-a239-48b9-8dae-f94be920ae1a-image-registry-private-configuration\") pod \"image-registry-65859cd4cb-2n5rm\" (UID: \"f8c1fa0c-a239-48b9-8dae-f94be920ae1a\") " pod="openshift-image-registry/image-registry-65859cd4cb-2n5rm" Jul 10 19:00:06.604506 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:06.601396 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f8c1fa0c-a239-48b9-8dae-f94be920ae1a-registry-tls\") pod \"image-registry-65859cd4cb-2n5rm\" (UID: \"f8c1fa0c-a239-48b9-8dae-f94be920ae1a\") " pod="openshift-image-registry/image-registry-65859cd4cb-2n5rm" Jul 10 19:00:06.604789 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:06.604656 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f8c1fa0c-a239-48b9-8dae-f94be920ae1a-installation-pull-secrets\") pod \"image-registry-65859cd4cb-2n5rm\" (UID: \"f8c1fa0c-a239-48b9-8dae-f94be920ae1a\") " pod="openshift-image-registry/image-registry-65859cd4cb-2n5rm" Jul 10 19:00:06.618025 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:06.617998 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f8c1fa0c-a239-48b9-8dae-f94be920ae1a-bound-sa-token\") pod \"image-registry-65859cd4cb-2n5rm\" (UID: \"f8c1fa0c-a239-48b9-8dae-f94be920ae1a\") " pod="openshift-image-registry/image-registry-65859cd4cb-2n5rm" Jul 10 19:00:06.618435 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:06.618410 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl2pp\" (UniqueName: \"kubernetes.io/projected/f8c1fa0c-a239-48b9-8dae-f94be920ae1a-kube-api-access-bl2pp\") pod \"image-registry-65859cd4cb-2n5rm\" (UID: \"f8c1fa0c-a239-48b9-8dae-f94be920ae1a\") " pod="openshift-image-registry/image-registry-65859cd4cb-2n5rm" Jul 10 19:00:06.638334 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:06.638302 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66d5548f5b-bvl7g" Jul 10 19:00:06.720786 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:06.720746 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65859cd4cb-2n5rm" Jul 10 19:00:07.455144 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:07.455057 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65859cd4cb-2n5rm"] Jul 10 19:00:07.457107 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:00:07.457083 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8c1fa0c_a239_48b9_8dae_f94be920ae1a.slice/crio-3e4ee6d381ccda7d39db0b223375aad4d2eef57fade1f85aec00d23d85a5d747 WatchSource:0}: Error finding container 3e4ee6d381ccda7d39db0b223375aad4d2eef57fade1f85aec00d23d85a5d747: Status 404 returned error can't find the container with id 3e4ee6d381ccda7d39db0b223375aad4d2eef57fade1f85aec00d23d85a5d747 Jul 10 19:00:07.470107 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:07.470082 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-66d5548f5b-bvl7g"] Jul 10 19:00:07.472787 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:00:07.472749 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca9cf55a_1bb3_4515_857a_3bd7f9c11795.slice/crio-52a7c60d463e9f15ff9286de2eeed224d4f8cec14f9948dc283bc232950eeec7 WatchSource:0}: Error finding container 52a7c60d463e9f15ff9286de2eeed224d4f8cec14f9948dc283bc232950eeec7: Status 404 returned error can't find the container with id 52a7c60d463e9f15ff9286de2eeed224d4f8cec14f9948dc283bc232950eeec7 Jul 10 19:00:08.025535 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:08.025495 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66d5548f5b-bvl7g" event={"ID":"ca9cf55a-1bb3-4515-857a-3bd7f9c11795","Type":"ContainerStarted","Data":"52a7c60d463e9f15ff9286de2eeed224d4f8cec14f9948dc283bc232950eeec7"} Jul 10 19:00:08.027420 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:08.027397 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5fqvm" event={"ID":"89e10ee7-fb7a-4cc2-9638-030cf5c0a305","Type":"ContainerStarted","Data":"ca11349c02376b4d83b3b7f4fe8d80c0201abe93c939bddd382b6e8dff44ba21"} Jul 10 19:00:08.028786 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:08.028765 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65859cd4cb-2n5rm" event={"ID":"f8c1fa0c-a239-48b9-8dae-f94be920ae1a","Type":"ContainerStarted","Data":"c3ff9db8792a92a47b37d88e9f66dbf3b07cd0fadb5ab98773ebb18673b00c17"} Jul 10 19:00:08.028887 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:08.028799 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65859cd4cb-2n5rm" event={"ID":"f8c1fa0c-a239-48b9-8dae-f94be920ae1a","Type":"ContainerStarted","Data":"3e4ee6d381ccda7d39db0b223375aad4d2eef57fade1f85aec00d23d85a5d747"} Jul 10 19:00:08.028934 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:08.028889 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-65859cd4cb-2n5rm" Jul 10 19:00:08.052906 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:08.052864 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5fqvm" podStartSLOduration=16.968213561 podStartE2EDuration="24.052850195s" totalImagesPullingTime="7.084636634s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:59:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:08.052518407 +0000 UTC m=+92.096660907" watchObservedRunningTime="2026-07-10 19:00:08.052850195 +0000 UTC m=+92.096992696" Jul 10 19:00:08.089911 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:08.089862 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-65859cd4cb-2n5rm" podStartSLOduration=2.089845635 podStartE2EDuration="2.089845635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:00:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:08.088899615 +0000 UTC m=+92.133042136" watchObservedRunningTime="2026-07-10 19:00:08.089845635 +0000 UTC m=+92.133988136" Jul 10 19:00:10.227512 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:10.227459 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/348b64e9-4db6-4d74-9a29-36436b02e1d3-service-ca-bundle\") pod \"router-default-8c8d8cb74-pbqx5\" (UID: \"348b64e9-4db6-4d74-9a29-36436b02e1d3\") " pod="openshift-ingress/router-default-8c8d8cb74-pbqx5" Jul 10 19:00:10.228199 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:10.228178 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/348b64e9-4db6-4d74-9a29-36436b02e1d3-service-ca-bundle\") pod \"router-default-8c8d8cb74-pbqx5\" (UID: \"348b64e9-4db6-4d74-9a29-36436b02e1d3\") " pod="openshift-ingress/router-default-8c8d8cb74-pbqx5" Jul 10 19:00:10.511064 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:10.511024 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-8c8d8cb74-pbqx5" Jul 10 19:00:11.399394 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:11.399364 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-8c8d8cb74-pbqx5"] Jul 10 19:00:11.403438 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:00:11.403415 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod348b64e9_4db6_4d74_9a29_36436b02e1d3.slice/crio-26220727057d85db3da59c3bddf5f19c0155f0c1fe65eb35113113f0190c9ce9 WatchSource:0}: Error finding container 26220727057d85db3da59c3bddf5f19c0155f0c1fe65eb35113113f0190c9ce9: Status 404 returned error can't find the container with id 26220727057d85db3da59c3bddf5f19c0155f0c1fe65eb35113113f0190c9ce9 Jul 10 19:00:12.049074 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:12.049037 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66d5548f5b-bvl7g" event={"ID":"ca9cf55a-1bb3-4515-857a-3bd7f9c11795","Type":"ContainerStarted","Data":"ce4deeaf6e22cf5389f60af26a6fe328e146cd59f838a5dcc8a155304219ee04"} Jul 10 19:00:12.049267 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:12.049236 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66d5548f5b-bvl7g" Jul 10 19:00:12.050639 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:12.050614 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-8c8d8cb74-pbqx5" event={"ID":"348b64e9-4db6-4d74-9a29-36436b02e1d3","Type":"ContainerStarted","Data":"d4c432074f3f5e50a5bd94bc7d5842f352550455bc234c83984e5d6b1ee8c708"} Jul 10 19:00:12.050797 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:12.050652 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-8c8d8cb74-pbqx5" event={"ID":"348b64e9-4db6-4d74-9a29-36436b02e1d3","Type":"ContainerStarted","Data":"26220727057d85db3da59c3bddf5f19c0155f0c1fe65eb35113113f0190c9ce9"} Jul 10 19:00:12.051148 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:12.051128 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66d5548f5b-bvl7g" Jul 10 19:00:12.069764 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:12.069722 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66d5548f5b-bvl7g" podStartSLOduration=2.201507073 podStartE2EDuration="6.069708643s" totalImagesPullingTime="3.86820157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:00:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:12.069415413 +0000 UTC m=+96.113557915" watchObservedRunningTime="2026-07-10 19:00:12.069708643 +0000 UTC m=+96.113851141" Jul 10 19:00:12.123479 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:12.123419 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-8c8d8cb74-pbqx5" podStartSLOduration=34.123402042 podStartE2EDuration="34.123402042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:59:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:12.094494399 +0000 UTC m=+96.138636927" watchObservedRunningTime="2026-07-10 19:00:12.123402042 +0000 UTC m=+96.167544540" Jul 10 19:00:12.512089 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:12.512058 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-8c8d8cb74-pbqx5" Jul 10 19:00:12.514646 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:12.514626 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-8c8d8cb74-pbqx5" Jul 10 19:00:13.054973 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:13.054942 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-8c8d8cb74-pbqx5" Jul 10 19:00:13.056074 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:13.056053 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-8c8d8cb74-pbqx5" Jul 10 19:00:16.631299 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:16.631261 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-d5ww8"] Jul 10 19:00:16.637221 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:16.637203 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-d5ww8" Jul 10 19:00:16.646416 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:16.646393 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 10 19:00:16.646536 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:16.646400 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 10 19:00:16.646536 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:16.646400 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 10 19:00:16.647835 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:16.647803 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 10 19:00:16.648116 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:16.648103 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-gjfmc\"" Jul 10 19:00:16.648501 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:16.648488 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 10 19:00:16.659434 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:16.659412 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-d5ww8"] Jul 10 19:00:16.774313 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:16.774276 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d6eebd0a-7281-40c4-81d0-9d1088043c1d-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-d5ww8\" (UID: \"d6eebd0a-7281-40c4-81d0-9d1088043c1d\") " pod="openshift-monitoring/prometheus-operator-64785b6675-d5ww8" Jul 10 19:00:16.774515 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:16.774325 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ed9ac220-276d-4864-b19b-17b8f69eaa98-metrics-tls\") pod \"dns-default-v5rql\" (UID: \"ed9ac220-276d-4864-b19b-17b8f69eaa98\") " pod="openshift-dns/dns-default-v5rql" Jul 10 19:00:16.774515 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:16.774396 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6eebd0a-7281-40c4-81d0-9d1088043c1d-metrics-client-ca\") pod \"prometheus-operator-64785b6675-d5ww8\" (UID: \"d6eebd0a-7281-40c4-81d0-9d1088043c1d\") " pod="openshift-monitoring/prometheus-operator-64785b6675-d5ww8" Jul 10 19:00:16.774515 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:16.774441 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d6eebd0a-7281-40c4-81d0-9d1088043c1d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-d5ww8\" (UID: \"d6eebd0a-7281-40c4-81d0-9d1088043c1d\") " pod="openshift-monitoring/prometheus-operator-64785b6675-d5ww8" Jul 10 19:00:16.774515 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:16.774473 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk925\" (UniqueName: \"kubernetes.io/projected/d6eebd0a-7281-40c4-81d0-9d1088043c1d-kube-api-access-jk925\") pod \"prometheus-operator-64785b6675-d5ww8\" (UID: \"d6eebd0a-7281-40c4-81d0-9d1088043c1d\") " pod="openshift-monitoring/prometheus-operator-64785b6675-d5ww8" Jul 10 19:00:16.776626 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:16.776594 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ed9ac220-276d-4864-b19b-17b8f69eaa98-metrics-tls\") pod \"dns-default-v5rql\" (UID: \"ed9ac220-276d-4864-b19b-17b8f69eaa98\") " pod="openshift-dns/dns-default-v5rql" Jul 10 19:00:16.875884 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:16.875834 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6eebd0a-7281-40c4-81d0-9d1088043c1d-metrics-client-ca\") pod \"prometheus-operator-64785b6675-d5ww8\" (UID: \"d6eebd0a-7281-40c4-81d0-9d1088043c1d\") " pod="openshift-monitoring/prometheus-operator-64785b6675-d5ww8" Jul 10 19:00:16.875884 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:16.875886 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d6eebd0a-7281-40c4-81d0-9d1088043c1d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-d5ww8\" (UID: \"d6eebd0a-7281-40c4-81d0-9d1088043c1d\") " pod="openshift-monitoring/prometheus-operator-64785b6675-d5ww8" Jul 10 19:00:16.876144 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:16.876013 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk925\" (UniqueName: \"kubernetes.io/projected/d6eebd0a-7281-40c4-81d0-9d1088043c1d-kube-api-access-jk925\") pod \"prometheus-operator-64785b6675-d5ww8\" (UID: \"d6eebd0a-7281-40c4-81d0-9d1088043c1d\") " pod="openshift-monitoring/prometheus-operator-64785b6675-d5ww8" Jul 10 19:00:16.876144 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:16.876105 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d6eebd0a-7281-40c4-81d0-9d1088043c1d-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-d5ww8\" (UID: \"d6eebd0a-7281-40c4-81d0-9d1088043c1d\") " pod="openshift-monitoring/prometheus-operator-64785b6675-d5ww8" Jul 10 19:00:16.876284 ip-10-0-135-67 kubenswrapper[2581]: E0710 19:00:16.876261 2581 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 10 19:00:16.876351 ip-10-0-135-67 kubenswrapper[2581]: E0710 19:00:16.876339 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6eebd0a-7281-40c4-81d0-9d1088043c1d-prometheus-operator-tls podName:d6eebd0a-7281-40c4-81d0-9d1088043c1d nodeName:}" failed. No retries permitted until 2026-07-10 19:00:17.37631929 +0000 UTC m=+101.420461774 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/d6eebd0a-7281-40c4-81d0-9d1088043c1d-prometheus-operator-tls") pod "prometheus-operator-64785b6675-d5ww8" (UID: "d6eebd0a-7281-40c4-81d0-9d1088043c1d") : secret "prometheus-operator-tls" not found Jul 10 19:00:16.876606 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:16.876590 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6eebd0a-7281-40c4-81d0-9d1088043c1d-metrics-client-ca\") pod \"prometheus-operator-64785b6675-d5ww8\" (UID: \"d6eebd0a-7281-40c4-81d0-9d1088043c1d\") " pod="openshift-monitoring/prometheus-operator-64785b6675-d5ww8" Jul 10 19:00:16.878223 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:16.878203 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d6eebd0a-7281-40c4-81d0-9d1088043c1d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-d5ww8\" (UID: \"d6eebd0a-7281-40c4-81d0-9d1088043c1d\") " pod="openshift-monitoring/prometheus-operator-64785b6675-d5ww8" Jul 10 19:00:16.890558 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:16.890529 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk925\" (UniqueName: \"kubernetes.io/projected/d6eebd0a-7281-40c4-81d0-9d1088043c1d-kube-api-access-jk925\") pod \"prometheus-operator-64785b6675-d5ww8\" (UID: \"d6eebd0a-7281-40c4-81d0-9d1088043c1d\") " pod="openshift-monitoring/prometheus-operator-64785b6675-d5ww8" Jul 10 19:00:17.065314 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:17.065219 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-s6z54\"" Jul 10 19:00:17.070810 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:17.070782 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-v5rql" Jul 10 19:00:17.203452 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:17.203423 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-v5rql"] Jul 10 19:00:17.205776 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:00:17.205740 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded9ac220_276d_4864_b19b_17b8f69eaa98.slice/crio-6e8d76f8ef2359e77612e8602498fff591f1b8c53e8699063ae1a88cba6b2e93 WatchSource:0}: Error finding container 6e8d76f8ef2359e77612e8602498fff591f1b8c53e8699063ae1a88cba6b2e93: Status 404 returned error can't find the container with id 6e8d76f8ef2359e77612e8602498fff591f1b8c53e8699063ae1a88cba6b2e93 Jul 10 19:00:17.380589 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:17.380505 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d6eebd0a-7281-40c4-81d0-9d1088043c1d-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-d5ww8\" (UID: \"d6eebd0a-7281-40c4-81d0-9d1088043c1d\") " pod="openshift-monitoring/prometheus-operator-64785b6675-d5ww8" Jul 10 19:00:17.382902 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:17.382884 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d6eebd0a-7281-40c4-81d0-9d1088043c1d-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-d5ww8\" (UID: \"d6eebd0a-7281-40c4-81d0-9d1088043c1d\") " pod="openshift-monitoring/prometheus-operator-64785b6675-d5ww8" Jul 10 19:00:17.545752 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:17.545722 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-d5ww8" Jul 10 19:00:17.673341 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:17.673251 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-d5ww8"] Jul 10 19:00:17.676166 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:00:17.676139 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6eebd0a_7281_40c4_81d0_9d1088043c1d.slice/crio-1743f8ad31b96a247be0570842d4f2fea3b0ef745fbbecf1de170fc6c65d8767 WatchSource:0}: Error finding container 1743f8ad31b96a247be0570842d4f2fea3b0ef745fbbecf1de170fc6c65d8767: Status 404 returned error can't find the container with id 1743f8ad31b96a247be0570842d4f2fea3b0ef745fbbecf1de170fc6c65d8767 Jul 10 19:00:17.961205 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:17.961120 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-4fmmt" Jul 10 19:00:18.070973 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:18.070929 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-d5ww8" event={"ID":"d6eebd0a-7281-40c4-81d0-9d1088043c1d","Type":"ContainerStarted","Data":"1743f8ad31b96a247be0570842d4f2fea3b0ef745fbbecf1de170fc6c65d8767"} Jul 10 19:00:18.072646 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:18.072606 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v5rql" event={"ID":"ed9ac220-276d-4864-b19b-17b8f69eaa98","Type":"ContainerStarted","Data":"6e8d76f8ef2359e77612e8602498fff591f1b8c53e8699063ae1a88cba6b2e93"} Jul 10 19:00:20.080489 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:20.080452 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-d5ww8" event={"ID":"d6eebd0a-7281-40c4-81d0-9d1088043c1d","Type":"ContainerStarted","Data":"e423d09f16d3f847cbe43898f07474d903c38f2261ab3380230c968e810eccae"} Jul 10 19:00:20.080924 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:20.080516 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-d5ww8" event={"ID":"d6eebd0a-7281-40c4-81d0-9d1088043c1d","Type":"ContainerStarted","Data":"dcb897de2dc9e134245a127375efc57fe31330e6a13d07fecfb0fa655f3dc6ec"} Jul 10 19:00:20.082160 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:20.082135 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v5rql" event={"ID":"ed9ac220-276d-4864-b19b-17b8f69eaa98","Type":"ContainerStarted","Data":"2a26a9d0a346186321e93b0aa463f97c58a44c09f13782e9feef16dfb41c62c9"} Jul 10 19:00:20.082293 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:20.082176 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v5rql" event={"ID":"ed9ac220-276d-4864-b19b-17b8f69eaa98","Type":"ContainerStarted","Data":"ef2630cee0745cb47c595734ca4d91055ba0fc6ea9431bbff9f40a1ac69215d4"} Jul 10 19:00:20.082293 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:20.082241 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-v5rql" Jul 10 19:00:20.100137 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:20.100095 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-d5ww8" podStartSLOduration=2.424189394 podStartE2EDuration="4.100078875s" totalImagesPullingTime="1.675889481s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:00:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:20.098831603 +0000 UTC m=+104.142974109" watchObservedRunningTime="2026-07-10 19:00:20.100078875 +0000 UTC m=+104.144221375" Jul 10 19:00:20.117761 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:20.117657 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-v5rql" podStartSLOduration=66.07329072 podStartE2EDuration="1m8.117642142s" totalImagesPullingTime="2.044351422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:59:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:20.117454505 +0000 UTC m=+104.161597006" watchObservedRunningTime="2026-07-10 19:00:20.117642142 +0000 UTC m=+104.161784642" Jul 10 19:00:22.017178 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:22.017143 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-p9h5z"] Jul 10 19:00:22.043515 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:22.043474 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-p9h5z" Jul 10 19:00:22.047103 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:22.046992 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 10 19:00:22.047103 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:22.047009 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gq9d6\"" Jul 10 19:00:22.047610 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:22.047593 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 10 19:00:22.047771 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:22.047756 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 10 19:00:22.223913 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:22.223881 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9c52bdca-eacb-4bb6-9305-ada12d71dede-node-exporter-accelerators-collector-config\") pod \"node-exporter-p9h5z\" (UID: \"9c52bdca-eacb-4bb6-9305-ada12d71dede\") " pod="openshift-monitoring/node-exporter-p9h5z" Jul 10 19:00:22.224074 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:22.223926 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c52bdca-eacb-4bb6-9305-ada12d71dede-sys\") pod \"node-exporter-p9h5z\" (UID: \"9c52bdca-eacb-4bb6-9305-ada12d71dede\") " pod="openshift-monitoring/node-exporter-p9h5z" Jul 10 19:00:22.224074 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:22.223956 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9c52bdca-eacb-4bb6-9305-ada12d71dede-node-exporter-wtmp\") pod \"node-exporter-p9h5z\" (UID: \"9c52bdca-eacb-4bb6-9305-ada12d71dede\") " pod="openshift-monitoring/node-exporter-p9h5z" Jul 10 19:00:22.224074 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:22.223983 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9c52bdca-eacb-4bb6-9305-ada12d71dede-node-exporter-tls\") pod \"node-exporter-p9h5z\" (UID: \"9c52bdca-eacb-4bb6-9305-ada12d71dede\") " pod="openshift-monitoring/node-exporter-p9h5z" Jul 10 19:00:22.224074 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:22.224005 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c52bdca-eacb-4bb6-9305-ada12d71dede-metrics-client-ca\") pod \"node-exporter-p9h5z\" (UID: \"9c52bdca-eacb-4bb6-9305-ada12d71dede\") " pod="openshift-monitoring/node-exporter-p9h5z" Jul 10 19:00:22.224074 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:22.224031 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c52bdca-eacb-4bb6-9305-ada12d71dede-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p9h5z\" (UID: \"9c52bdca-eacb-4bb6-9305-ada12d71dede\") " pod="openshift-monitoring/node-exporter-p9h5z" Jul 10 19:00:22.224074 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:22.224067 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9c52bdca-eacb-4bb6-9305-ada12d71dede-node-exporter-textfile\") pod \"node-exporter-p9h5z\" (UID: \"9c52bdca-eacb-4bb6-9305-ada12d71dede\") " pod="openshift-monitoring/node-exporter-p9h5z" Jul 10 19:00:22.224314 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:22.224094 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4w94\" (UniqueName: \"kubernetes.io/projected/9c52bdca-eacb-4bb6-9305-ada12d71dede-kube-api-access-s4w94\") pod \"node-exporter-p9h5z\" (UID: \"9c52bdca-eacb-4bb6-9305-ada12d71dede\") " pod="openshift-monitoring/node-exporter-p9h5z" Jul 10 19:00:22.224314 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:22.224134 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9c52bdca-eacb-4bb6-9305-ada12d71dede-root\") pod \"node-exporter-p9h5z\" (UID: \"9c52bdca-eacb-4bb6-9305-ada12d71dede\") " pod="openshift-monitoring/node-exporter-p9h5z" Jul 10 19:00:22.324845 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:22.324749 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4w94\" (UniqueName: \"kubernetes.io/projected/9c52bdca-eacb-4bb6-9305-ada12d71dede-kube-api-access-s4w94\") pod \"node-exporter-p9h5z\" (UID: \"9c52bdca-eacb-4bb6-9305-ada12d71dede\") " pod="openshift-monitoring/node-exporter-p9h5z" Jul 10 19:00:22.324845 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:22.324797 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9c52bdca-eacb-4bb6-9305-ada12d71dede-root\") pod \"node-exporter-p9h5z\" (UID: \"9c52bdca-eacb-4bb6-9305-ada12d71dede\") " pod="openshift-monitoring/node-exporter-p9h5z" Jul 10 19:00:22.324845 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:22.324843 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9c52bdca-eacb-4bb6-9305-ada12d71dede-node-exporter-accelerators-collector-config\") pod \"node-exporter-p9h5z\" (UID: \"9c52bdca-eacb-4bb6-9305-ada12d71dede\") " pod="openshift-monitoring/node-exporter-p9h5z" Jul 10 19:00:22.325122 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:22.324860 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c52bdca-eacb-4bb6-9305-ada12d71dede-sys\") pod \"node-exporter-p9h5z\" (UID: \"9c52bdca-eacb-4bb6-9305-ada12d71dede\") " pod="openshift-monitoring/node-exporter-p9h5z" Jul 10 19:00:22.325122 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:22.324881 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9c52bdca-eacb-4bb6-9305-ada12d71dede-node-exporter-wtmp\") pod \"node-exporter-p9h5z\" (UID: \"9c52bdca-eacb-4bb6-9305-ada12d71dede\") " pod="openshift-monitoring/node-exporter-p9h5z" Jul 10 19:00:22.325122 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:22.324904 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9c52bdca-eacb-4bb6-9305-ada12d71dede-node-exporter-tls\") pod \"node-exporter-p9h5z\" (UID: \"9c52bdca-eacb-4bb6-9305-ada12d71dede\") " pod="openshift-monitoring/node-exporter-p9h5z" Jul 10 19:00:22.325122 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:22.324925 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c52bdca-eacb-4bb6-9305-ada12d71dede-metrics-client-ca\") pod \"node-exporter-p9h5z\" (UID: \"9c52bdca-eacb-4bb6-9305-ada12d71dede\") " pod="openshift-monitoring/node-exporter-p9h5z" Jul 10 19:00:22.325122 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:22.324960 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9c52bdca-eacb-4bb6-9305-ada12d71dede-root\") pod \"node-exporter-p9h5z\" (UID: \"9c52bdca-eacb-4bb6-9305-ada12d71dede\") " pod="openshift-monitoring/node-exporter-p9h5z" Jul 10 19:00:22.325122 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:22.324994 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c52bdca-eacb-4bb6-9305-ada12d71dede-sys\") pod \"node-exporter-p9h5z\" (UID: \"9c52bdca-eacb-4bb6-9305-ada12d71dede\") " pod="openshift-monitoring/node-exporter-p9h5z" Jul 10 19:00:22.325122 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:22.325035 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c52bdca-eacb-4bb6-9305-ada12d71dede-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p9h5z\" (UID: \"9c52bdca-eacb-4bb6-9305-ada12d71dede\") " pod="openshift-monitoring/node-exporter-p9h5z" Jul 10 19:00:22.325122 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:22.325075 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9c52bdca-eacb-4bb6-9305-ada12d71dede-node-exporter-wtmp\") pod \"node-exporter-p9h5z\" (UID: \"9c52bdca-eacb-4bb6-9305-ada12d71dede\") " pod="openshift-monitoring/node-exporter-p9h5z" Jul 10 19:00:22.325122 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:22.325080 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9c52bdca-eacb-4bb6-9305-ada12d71dede-node-exporter-textfile\") pod \"node-exporter-p9h5z\" (UID: \"9c52bdca-eacb-4bb6-9305-ada12d71dede\") " pod="openshift-monitoring/node-exporter-p9h5z" Jul 10 19:00:22.325448 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:22.325353 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9c52bdca-eacb-4bb6-9305-ada12d71dede-node-exporter-textfile\") pod \"node-exporter-p9h5z\" (UID: \"9c52bdca-eacb-4bb6-9305-ada12d71dede\") " pod="openshift-monitoring/node-exporter-p9h5z" Jul 10 19:00:22.325581 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:22.325560 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c52bdca-eacb-4bb6-9305-ada12d71dede-metrics-client-ca\") pod \"node-exporter-p9h5z\" (UID: \"9c52bdca-eacb-4bb6-9305-ada12d71dede\") " pod="openshift-monitoring/node-exporter-p9h5z" Jul 10 19:00:22.325639 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:22.325564 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9c52bdca-eacb-4bb6-9305-ada12d71dede-node-exporter-accelerators-collector-config\") pod \"node-exporter-p9h5z\" (UID: \"9c52bdca-eacb-4bb6-9305-ada12d71dede\") " pod="openshift-monitoring/node-exporter-p9h5z" Jul 10 19:00:22.327450 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:22.327426 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9c52bdca-eacb-4bb6-9305-ada12d71dede-node-exporter-tls\") pod \"node-exporter-p9h5z\" (UID: \"9c52bdca-eacb-4bb6-9305-ada12d71dede\") " pod="openshift-monitoring/node-exporter-p9h5z" Jul 10 19:00:22.327561 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:22.327457 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c52bdca-eacb-4bb6-9305-ada12d71dede-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p9h5z\" (UID: \"9c52bdca-eacb-4bb6-9305-ada12d71dede\") " pod="openshift-monitoring/node-exporter-p9h5z" Jul 10 19:00:22.333958 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:22.333938 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4w94\" (UniqueName: \"kubernetes.io/projected/9c52bdca-eacb-4bb6-9305-ada12d71dede-kube-api-access-s4w94\") pod \"node-exporter-p9h5z\" (UID: \"9c52bdca-eacb-4bb6-9305-ada12d71dede\") " pod="openshift-monitoring/node-exporter-p9h5z" Jul 10 19:00:22.353043 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:22.353018 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-p9h5z" Jul 10 19:00:22.394333 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:00:22.394306 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c52bdca_eacb_4bb6_9305_ada12d71dede.slice/crio-c08a0e68b488433b7b43ed0c6a361c12a7b1c61251ec8d4fab189353498a4faf WatchSource:0}: Error finding container c08a0e68b488433b7b43ed0c6a361c12a7b1c61251ec8d4fab189353498a4faf: Status 404 returned error can't find the container with id c08a0e68b488433b7b43ed0c6a361c12a7b1c61251ec8d4fab189353498a4faf Jul 10 19:00:23.094447 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:23.094404 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p9h5z" event={"ID":"9c52bdca-eacb-4bb6-9305-ada12d71dede","Type":"ContainerStarted","Data":"c08a0e68b488433b7b43ed0c6a361c12a7b1c61251ec8d4fab189353498a4faf"} Jul 10 19:00:24.099408 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:24.099371 2581 generic.go:356] "Generic (PLEG): container finished" podID="9c52bdca-eacb-4bb6-9305-ada12d71dede" containerID="e44a3db60f8562eee8fb3cc876cb017465fa057be2c2b73627e094e62dd26a7c" exitCode=0 Jul 10 19:00:24.099838 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:24.099470 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p9h5z" event={"ID":"9c52bdca-eacb-4bb6-9305-ada12d71dede","Type":"ContainerDied","Data":"e44a3db60f8562eee8fb3cc876cb017465fa057be2c2b73627e094e62dd26a7c"} Jul 10 19:00:24.989011 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:24.988981 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7f747dbb6f-gjdd7"] Jul 10 19:00:24.991677 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:24.991657 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7f747dbb6f-gjdd7" Jul 10 19:00:24.994304 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:24.994252 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-a96p88br1nffk\"" Jul 10 19:00:24.994513 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:24.994497 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 10 19:00:24.994568 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:24.994513 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 10 19:00:24.994568 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:24.994554 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 10 19:00:24.994912 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:24.994899 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-4p54d\"" Jul 10 19:00:24.994957 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:24.994938 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 10 19:00:24.995008 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:24.994939 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 10 19:00:25.008341 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:25.008314 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7f747dbb6f-gjdd7"] Jul 10 19:00:25.109206 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:25.109173 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p9h5z" event={"ID":"9c52bdca-eacb-4bb6-9305-ada12d71dede","Type":"ContainerStarted","Data":"b42e70afe3371d950557aca067e5cef15c1daa5b7c70b0419419a373f244a2d5"} Jul 10 19:00:25.109573 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:25.109261 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p9h5z" event={"ID":"9c52bdca-eacb-4bb6-9305-ada12d71dede","Type":"ContainerStarted","Data":"1bee556dc045d63942a4f78afe9e03dd917b456cd333970881c92d91151a6fdb"} Jul 10 19:00:25.131377 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:25.131323 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-p9h5z" podStartSLOduration=2.88918953 podStartE2EDuration="4.131306411s" totalImagesPullingTime="1.177381151s" totalInitContainerRuntime="64.73573ms" isStatefulPod=true podCreationTimestamp="2026-07-10 19:00:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:25.130202328 +0000 UTC m=+109.174344840" watchObservedRunningTime="2026-07-10 19:00:25.131306411 +0000 UTC m=+109.175448912" Jul 10 19:00:25.148508 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:25.148478 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85-metrics-client-ca\") pod \"thanos-querier-7f747dbb6f-gjdd7\" (UID: \"3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85\") " pod="openshift-monitoring/thanos-querier-7f747dbb6f-gjdd7" Jul 10 19:00:25.148660 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:25.148517 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpnpj\" (UniqueName: \"kubernetes.io/projected/3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85-kube-api-access-xpnpj\") pod \"thanos-querier-7f747dbb6f-gjdd7\" (UID: \"3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85\") " pod="openshift-monitoring/thanos-querier-7f747dbb6f-gjdd7" Jul 10 19:00:25.148660 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:25.148537 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7f747dbb6f-gjdd7\" (UID: \"3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85\") " pod="openshift-monitoring/thanos-querier-7f747dbb6f-gjdd7" Jul 10 19:00:25.148660 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:25.148557 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7f747dbb6f-gjdd7\" (UID: \"3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85\") " pod="openshift-monitoring/thanos-querier-7f747dbb6f-gjdd7" Jul 10 19:00:25.148660 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:25.148614 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7f747dbb6f-gjdd7\" (UID: \"3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85\") " pod="openshift-monitoring/thanos-querier-7f747dbb6f-gjdd7" Jul 10 19:00:25.148841 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:25.148664 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7f747dbb6f-gjdd7\" (UID: \"3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85\") " pod="openshift-monitoring/thanos-querier-7f747dbb6f-gjdd7" Jul 10 19:00:25.148841 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:25.148712 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85-secret-thanos-querier-tls\") pod \"thanos-querier-7f747dbb6f-gjdd7\" (UID: \"3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85\") " pod="openshift-monitoring/thanos-querier-7f747dbb6f-gjdd7" Jul 10 19:00:25.148841 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:25.148762 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85-secret-grpc-tls\") pod \"thanos-querier-7f747dbb6f-gjdd7\" (UID: \"3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85\") " pod="openshift-monitoring/thanos-querier-7f747dbb6f-gjdd7" Jul 10 19:00:25.249357 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:25.249327 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85-metrics-client-ca\") pod \"thanos-querier-7f747dbb6f-gjdd7\" (UID: \"3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85\") " pod="openshift-monitoring/thanos-querier-7f747dbb6f-gjdd7" Jul 10 19:00:25.249522 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:25.249418 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpnpj\" (UniqueName: \"kubernetes.io/projected/3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85-kube-api-access-xpnpj\") pod \"thanos-querier-7f747dbb6f-gjdd7\" (UID: \"3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85\") " pod="openshift-monitoring/thanos-querier-7f747dbb6f-gjdd7" Jul 10 19:00:25.249522 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:25.249450 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7f747dbb6f-gjdd7\" (UID: \"3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85\") " pod="openshift-monitoring/thanos-querier-7f747dbb6f-gjdd7" Jul 10 19:00:25.249522 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:25.249481 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7f747dbb6f-gjdd7\" (UID: \"3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85\") " pod="openshift-monitoring/thanos-querier-7f747dbb6f-gjdd7" Jul 10 19:00:25.249522 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:25.249513 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7f747dbb6f-gjdd7\" (UID: \"3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85\") " pod="openshift-monitoring/thanos-querier-7f747dbb6f-gjdd7" Jul 10 19:00:25.249744 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:25.249539 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7f747dbb6f-gjdd7\" (UID: \"3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85\") " pod="openshift-monitoring/thanos-querier-7f747dbb6f-gjdd7" Jul 10 19:00:25.249744 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:25.249585 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85-secret-thanos-querier-tls\") pod \"thanos-querier-7f747dbb6f-gjdd7\" (UID: \"3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85\") " pod="openshift-monitoring/thanos-querier-7f747dbb6f-gjdd7" Jul 10 19:00:25.249744 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:25.249712 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85-secret-grpc-tls\") pod \"thanos-querier-7f747dbb6f-gjdd7\" (UID: \"3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85\") " pod="openshift-monitoring/thanos-querier-7f747dbb6f-gjdd7" Jul 10 19:00:25.250191 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:25.250164 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85-metrics-client-ca\") pod \"thanos-querier-7f747dbb6f-gjdd7\" (UID: \"3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85\") " pod="openshift-monitoring/thanos-querier-7f747dbb6f-gjdd7" Jul 10 19:00:25.252590 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:25.252564 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7f747dbb6f-gjdd7\" (UID: \"3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85\") " pod="openshift-monitoring/thanos-querier-7f747dbb6f-gjdd7" Jul 10 19:00:25.252590 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:25.252583 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85-secret-grpc-tls\") pod \"thanos-querier-7f747dbb6f-gjdd7\" (UID: \"3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85\") " pod="openshift-monitoring/thanos-querier-7f747dbb6f-gjdd7" Jul 10 19:00:25.252791 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:25.252565 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85-secret-thanos-querier-tls\") pod \"thanos-querier-7f747dbb6f-gjdd7\" (UID: \"3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85\") " pod="openshift-monitoring/thanos-querier-7f747dbb6f-gjdd7" Jul 10 19:00:25.252791 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:25.252715 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7f747dbb6f-gjdd7\" (UID: \"3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85\") " pod="openshift-monitoring/thanos-querier-7f747dbb6f-gjdd7" Jul 10 19:00:25.252791 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:25.252725 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7f747dbb6f-gjdd7\" (UID: \"3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85\") " pod="openshift-monitoring/thanos-querier-7f747dbb6f-gjdd7" Jul 10 19:00:25.253170 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:25.253141 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7f747dbb6f-gjdd7\" (UID: \"3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85\") " pod="openshift-monitoring/thanos-querier-7f747dbb6f-gjdd7" Jul 10 19:00:25.261885 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:25.261864 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpnpj\" (UniqueName: \"kubernetes.io/projected/3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85-kube-api-access-xpnpj\") pod \"thanos-querier-7f747dbb6f-gjdd7\" (UID: \"3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85\") " pod="openshift-monitoring/thanos-querier-7f747dbb6f-gjdd7" Jul 10 19:00:25.301411 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:25.301379 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7f747dbb6f-gjdd7" Jul 10 19:00:25.436620 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:25.436587 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7f747dbb6f-gjdd7"] Jul 10 19:00:25.439269 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:00:25.439241 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3edc6eb7_d84c_4a16_bc7a_ea3bf88d3b85.slice/crio-bba15d671d382d98086753cee03c349a4a55ae8ae9d4b69ef1947538871d3a36 WatchSource:0}: Error finding container bba15d671d382d98086753cee03c349a4a55ae8ae9d4b69ef1947538871d3a36: Status 404 returned error can't find the container with id bba15d671d382d98086753cee03c349a4a55ae8ae9d4b69ef1947538871d3a36 Jul 10 19:00:25.659325 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:25.659252 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-rhf5g"] Jul 10 19:00:25.662150 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:25.662121 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-rhf5g" Jul 10 19:00:25.664715 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:25.664694 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 10 19:00:25.664816 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:25.664755 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 10 19:00:25.664816 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:25.664804 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-d49ld\"" Jul 10 19:00:25.672543 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:25.672506 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-rhf5g"] Jul 10 19:00:25.856196 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:25.856162 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmlcr\" (UniqueName: \"kubernetes.io/projected/ad100d8d-a2d8-4c86-8b2c-2aecee40bfbf-kube-api-access-zmlcr\") pod \"downloads-68c9cbb497-rhf5g\" (UID: \"ad100d8d-a2d8-4c86-8b2c-2aecee40bfbf\") " pod="openshift-console/downloads-68c9cbb497-rhf5g" Jul 10 19:00:25.957064 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:25.956977 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmlcr\" (UniqueName: \"kubernetes.io/projected/ad100d8d-a2d8-4c86-8b2c-2aecee40bfbf-kube-api-access-zmlcr\") pod \"downloads-68c9cbb497-rhf5g\" (UID: \"ad100d8d-a2d8-4c86-8b2c-2aecee40bfbf\") " pod="openshift-console/downloads-68c9cbb497-rhf5g" Jul 10 19:00:25.965057 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:25.965026 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmlcr\" (UniqueName: \"kubernetes.io/projected/ad100d8d-a2d8-4c86-8b2c-2aecee40bfbf-kube-api-access-zmlcr\") pod \"downloads-68c9cbb497-rhf5g\" (UID: \"ad100d8d-a2d8-4c86-8b2c-2aecee40bfbf\") " pod="openshift-console/downloads-68c9cbb497-rhf5g" Jul 10 19:00:25.970764 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:25.970746 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-rhf5g" Jul 10 19:00:26.113326 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:26.113263 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-rhf5g"] Jul 10 19:00:26.114491 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:26.114441 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f747dbb6f-gjdd7" event={"ID":"3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85","Type":"ContainerStarted","Data":"bba15d671d382d98086753cee03c349a4a55ae8ae9d4b69ef1947538871d3a36"} Jul 10 19:00:26.116865 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:00:26.116830 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad100d8d_a2d8_4c86_8b2c_2aecee40bfbf.slice/crio-b08448789fae012a5f07cbee51c7e463e3689ec3add38da451b387a24f49c5d8 WatchSource:0}: Error finding container b08448789fae012a5f07cbee51c7e463e3689ec3add38da451b387a24f49c5d8: Status 404 returned error can't find the container with id b08448789fae012a5f07cbee51c7e463e3689ec3add38da451b387a24f49c5d8 Jul 10 19:00:26.728383 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:26.728343 2581 patch_prober.go:28] interesting pod/image-registry-65859cd4cb-2n5rm container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 10 19:00:26.728580 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:26.728406 2581 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-65859cd4cb-2n5rm" podUID="f8c1fa0c-a239-48b9-8dae-f94be920ae1a" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 19:00:26.746234 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:26.745571 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-ftv8l"] Jul 10 19:00:26.748489 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:26.748457 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-ftv8l" Jul 10 19:00:26.751178 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:26.751137 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-26gn8\"" Jul 10 19:00:26.754420 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:26.751359 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 10 19:00:26.757569 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:26.757533 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-ftv8l"] Jul 10 19:00:26.866057 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:26.866019 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9769ce8f-d048-4fd4-b4dd-3a2d636859f9-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-ftv8l\" (UID: \"9769ce8f-d048-4fd4-b4dd-3a2d636859f9\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-ftv8l" Jul 10 19:00:26.866313 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:26.866073 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-576hq\" (UniqueName: \"kubernetes.io/projected/9769ce8f-d048-4fd4-b4dd-3a2d636859f9-kube-api-access-576hq\") pod \"monitoring-plugin-5f68df7f58-ftv8l\" (UID: \"9769ce8f-d048-4fd4-b4dd-3a2d636859f9\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-ftv8l" Jul 10 19:00:26.967383 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:26.967348 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9769ce8f-d048-4fd4-b4dd-3a2d636859f9-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-ftv8l\" (UID: \"9769ce8f-d048-4fd4-b4dd-3a2d636859f9\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-ftv8l" Jul 10 19:00:26.967575 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:26.967395 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-576hq\" (UniqueName: \"kubernetes.io/projected/9769ce8f-d048-4fd4-b4dd-3a2d636859f9-kube-api-access-576hq\") pod \"monitoring-plugin-5f68df7f58-ftv8l\" (UID: \"9769ce8f-d048-4fd4-b4dd-3a2d636859f9\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-ftv8l" Jul 10 19:00:26.970323 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:26.970298 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9769ce8f-d048-4fd4-b4dd-3a2d636859f9-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-ftv8l\" (UID: \"9769ce8f-d048-4fd4-b4dd-3a2d636859f9\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-ftv8l" Jul 10 19:00:26.977015 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:26.976886 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-576hq\" (UniqueName: \"kubernetes.io/projected/9769ce8f-d048-4fd4-b4dd-3a2d636859f9-kube-api-access-576hq\") pod \"monitoring-plugin-5f68df7f58-ftv8l\" (UID: \"9769ce8f-d048-4fd4-b4dd-3a2d636859f9\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-ftv8l" Jul 10 19:00:27.061535 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:27.061494 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-ftv8l" Jul 10 19:00:27.119762 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:27.119625 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-rhf5g" event={"ID":"ad100d8d-a2d8-4c86-8b2c-2aecee40bfbf","Type":"ContainerStarted","Data":"b08448789fae012a5f07cbee51c7e463e3689ec3add38da451b387a24f49c5d8"} Jul 10 19:00:27.530277 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:27.530242 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-ftv8l"] Jul 10 19:00:27.539161 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:00:27.538342 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9769ce8f_d048_4fd4_b4dd_3a2d636859f9.slice/crio-0270cbb8b261ebac267b2e57515a024465e411128b1e07c290e704f4f9524a2f WatchSource:0}: Error finding container 0270cbb8b261ebac267b2e57515a024465e411128b1e07c290e704f4f9524a2f: Status 404 returned error can't find the container with id 0270cbb8b261ebac267b2e57515a024465e411128b1e07c290e704f4f9524a2f Jul 10 19:00:28.131514 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:28.131475 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-ftv8l" event={"ID":"9769ce8f-d048-4fd4-b4dd-3a2d636859f9","Type":"ContainerStarted","Data":"0270cbb8b261ebac267b2e57515a024465e411128b1e07c290e704f4f9524a2f"} Jul 10 19:00:28.134889 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:28.134848 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f747dbb6f-gjdd7" event={"ID":"3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85","Type":"ContainerStarted","Data":"ecc71fa77c76ad0c9475f3979e871baeb65cc0da5f1e3f00b4cdf102ad31a90f"} Jul 10 19:00:28.135010 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:28.134931 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f747dbb6f-gjdd7" event={"ID":"3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85","Type":"ContainerStarted","Data":"228f437f0a991c01bf078235c48f86341864abd313bd92ec2a830bb354b039a8"} Jul 10 19:00:28.135010 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:28.134973 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f747dbb6f-gjdd7" event={"ID":"3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85","Type":"ContainerStarted","Data":"455cf0fa48e8982c64884e6f10936eb37f45ec358b2e12a29780c2dc01746cb0"} Jul 10 19:00:29.038142 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:29.038098 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-65859cd4cb-2n5rm" Jul 10 19:00:29.142721 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:29.142675 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f747dbb6f-gjdd7" event={"ID":"3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85","Type":"ContainerStarted","Data":"b705fa8d915e5c5260b2f04999ed1377d73870308a963b4f3e80eba2ee0040ef"} Jul 10 19:00:29.144285 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:29.144253 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-ftv8l" event={"ID":"9769ce8f-d048-4fd4-b4dd-3a2d636859f9","Type":"ContainerStarted","Data":"ad7f90651aa5a3bb63509776ff000cb96beeea4a82afedcad0ba1fd7e84b38f5"} Jul 10 19:00:29.144478 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:29.144455 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-ftv8l" Jul 10 19:00:29.150223 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:29.150168 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-ftv8l" Jul 10 19:00:29.167105 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:29.167063 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-ftv8l" podStartSLOduration=1.685649562 podStartE2EDuration="3.16705034s" totalImagesPullingTime="1.481400778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:00:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:29.165276271 +0000 UTC m=+113.209418789" watchObservedRunningTime="2026-07-10 19:00:29.16705034 +0000 UTC m=+113.211192866" Jul 10 19:00:30.088931 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:30.088903 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-v5rql" Jul 10 19:00:30.152892 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:30.152808 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f747dbb6f-gjdd7" event={"ID":"3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85","Type":"ContainerStarted","Data":"5f7901d547f0c0e765f0405c7b413f7c171354e5f910623ddd92b6e3ed306bfa"} Jul 10 19:00:30.153340 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:30.152900 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f747dbb6f-gjdd7" event={"ID":"3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85","Type":"ContainerStarted","Data":"31a36e5e34802b36ef080fb617ec9fbb5107ad69b9b20c612b51fda282355a71"} Jul 10 19:00:30.153340 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:30.153234 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7f747dbb6f-gjdd7" Jul 10 19:00:30.174213 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:30.174158 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7f747dbb6f-gjdd7" podStartSLOduration=2.820486469 podStartE2EDuration="6.174135562s" totalImagesPullingTime="3.353649093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:00:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:30.173124639 +0000 UTC m=+114.217267162" watchObservedRunningTime="2026-07-10 19:00:30.174135562 +0000 UTC m=+114.218278064" Jul 10 19:00:36.164717 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:36.164673 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7f747dbb6f-gjdd7" Jul 10 19:00:44.217586 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:44.217537 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-rhf5g" event={"ID":"ad100d8d-a2d8-4c86-8b2c-2aecee40bfbf","Type":"ContainerStarted","Data":"6e0d671df4f0a4dce882be569f952bcd660cc6ba91fa117d1b365edecd85bf51"} Jul 10 19:00:44.218221 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:44.217730 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-rhf5g" Jul 10 19:00:44.233665 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:44.233640 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-rhf5g" Jul 10 19:00:44.243939 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:44.243888 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-rhf5g" podStartSLOduration=2.097424775 podStartE2EDuration="19.243872419s" totalImagesPullingTime="17.146447644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:00:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:44.241901353 +0000 UTC m=+128.286043881" watchObservedRunningTime="2026-07-10 19:00:44.243872419 +0000 UTC m=+128.288014920" Jul 10 19:00:45.330183 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:45.330134 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71f50128-1d81-4578-8758-cc4b92443331-metrics-certs\") pod \"network-metrics-daemon-brfp2\" (UID: \"71f50128-1d81-4578-8758-cc4b92443331\") " pod="openshift-multus/network-metrics-daemon-brfp2" Jul 10 19:00:45.332945 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:45.332915 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71f50128-1d81-4578-8758-cc4b92443331-metrics-certs\") pod \"network-metrics-daemon-brfp2\" (UID: \"71f50128-1d81-4578-8758-cc4b92443331\") " pod="openshift-multus/network-metrics-daemon-brfp2" Jul 10 19:00:45.460583 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:45.460550 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mzc76\"" Jul 10 19:00:45.468527 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:45.468501 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-brfp2" Jul 10 19:00:45.657597 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:45.657564 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-brfp2"] Jul 10 19:00:45.660840 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:00:45.660811 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71f50128_1d81_4578_8758_cc4b92443331.slice/crio-d93b30762599b89ec86b9f9fecf4e95badf611da7e7cd882b9182998c246a7b3 WatchSource:0}: Error finding container d93b30762599b89ec86b9f9fecf4e95badf611da7e7cd882b9182998c246a7b3: Status 404 returned error can't find the container with id d93b30762599b89ec86b9f9fecf4e95badf611da7e7cd882b9182998c246a7b3 Jul 10 19:00:46.228148 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:46.228065 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-brfp2" event={"ID":"71f50128-1d81-4578-8758-cc4b92443331","Type":"ContainerStarted","Data":"d93b30762599b89ec86b9f9fecf4e95badf611da7e7cd882b9182998c246a7b3"} Jul 10 19:00:48.239315 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:48.239272 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-brfp2" event={"ID":"71f50128-1d81-4578-8758-cc4b92443331","Type":"ContainerStarted","Data":"2ed03834dd372ea5902d4c9b9dec6414e23a18e13ed6a83cb6996476e43241fa"} Jul 10 19:00:48.239905 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:48.239338 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-brfp2" event={"ID":"71f50128-1d81-4578-8758-cc4b92443331","Type":"ContainerStarted","Data":"5848cae3c50bed2681c713cde21945a43b4c7ec8d7016dbd639c42596323678d"} Jul 10 19:00:48.258864 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:00:48.258810 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-brfp2" podStartSLOduration=129.66520728 podStartE2EDuration="2m11.258790003s" totalImagesPullingTime="1.593582723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:48.257029689 +0000 UTC m=+132.301172215" watchObservedRunningTime="2026-07-10 19:00:48.258790003 +0000 UTC m=+132.302932504" Jul 10 19:03:36.398188 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:03:36.398153 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69hnr_7f479cec-2fb8-45c4-ae7d-073194a5dede/ovn-acl-logging/0.log" Jul 10 19:03:36.403380 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:03:36.403355 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69hnr_7f479cec-2fb8-45c4-ae7d-073194a5dede/ovn-acl-logging/0.log" Jul 10 19:03:36.406546 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:03:36.406528 2581 kubelet.go:1648] "Image garbage collection succeeded" Jul 10 19:03:52.654823 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:03:52.654789 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pn49"] Jul 10 19:03:52.656993 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:03:52.656977 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pn49" Jul 10 19:03:52.659672 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:03:52.659647 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 10 19:03:52.659831 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:03:52.659656 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 10 19:03:52.659831 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:03:52.659707 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-b9hpz\"" Jul 10 19:03:52.666145 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:03:52.666118 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pn49"] Jul 10 19:03:52.734993 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:03:52.734953 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr7f8\" (UniqueName: \"kubernetes.io/projected/81f793e8-2ce7-46a9-8d5e-eff3b412a34f-kube-api-access-kr7f8\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pn49\" (UID: \"81f793e8-2ce7-46a9-8d5e-eff3b412a34f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pn49" Jul 10 19:03:52.735161 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:03:52.735003 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/81f793e8-2ce7-46a9-8d5e-eff3b412a34f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pn49\" (UID: \"81f793e8-2ce7-46a9-8d5e-eff3b412a34f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pn49" Jul 10 19:03:52.735161 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:03:52.735099 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/81f793e8-2ce7-46a9-8d5e-eff3b412a34f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pn49\" (UID: \"81f793e8-2ce7-46a9-8d5e-eff3b412a34f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pn49" Jul 10 19:03:52.835892 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:03:52.835857 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/81f793e8-2ce7-46a9-8d5e-eff3b412a34f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pn49\" (UID: \"81f793e8-2ce7-46a9-8d5e-eff3b412a34f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pn49" Jul 10 19:03:52.835979 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:03:52.835920 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/81f793e8-2ce7-46a9-8d5e-eff3b412a34f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pn49\" (UID: \"81f793e8-2ce7-46a9-8d5e-eff3b412a34f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pn49" Jul 10 19:03:52.835979 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:03:52.835948 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr7f8\" (UniqueName: \"kubernetes.io/projected/81f793e8-2ce7-46a9-8d5e-eff3b412a34f-kube-api-access-kr7f8\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pn49\" (UID: \"81f793e8-2ce7-46a9-8d5e-eff3b412a34f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pn49" Jul 10 19:03:52.836280 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:03:52.836257 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/81f793e8-2ce7-46a9-8d5e-eff3b412a34f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pn49\" (UID: \"81f793e8-2ce7-46a9-8d5e-eff3b412a34f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pn49" Jul 10 19:03:52.836317 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:03:52.836281 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/81f793e8-2ce7-46a9-8d5e-eff3b412a34f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pn49\" (UID: \"81f793e8-2ce7-46a9-8d5e-eff3b412a34f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pn49" Jul 10 19:03:52.844323 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:03:52.844292 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr7f8\" (UniqueName: \"kubernetes.io/projected/81f793e8-2ce7-46a9-8d5e-eff3b412a34f-kube-api-access-kr7f8\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pn49\" (UID: \"81f793e8-2ce7-46a9-8d5e-eff3b412a34f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pn49" Jul 10 19:03:52.966879 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:03:52.966789 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pn49" Jul 10 19:03:53.093842 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:03:53.093810 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pn49"] Jul 10 19:03:53.099157 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:03:53.099121 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f793e8_2ce7_46a9_8d5e_eff3b412a34f.slice/crio-78561cc0cbfa688b6dcb0dccf035ae5c06d41bc8f4284e7725c852503377e9a5 WatchSource:0}: Error finding container 78561cc0cbfa688b6dcb0dccf035ae5c06d41bc8f4284e7725c852503377e9a5: Status 404 returned error can't find the container with id 78561cc0cbfa688b6dcb0dccf035ae5c06d41bc8f4284e7725c852503377e9a5 Jul 10 19:03:53.101741 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:03:53.101715 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 19:03:53.907945 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:03:53.907901 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pn49" event={"ID":"81f793e8-2ce7-46a9-8d5e-eff3b412a34f","Type":"ContainerStarted","Data":"78561cc0cbfa688b6dcb0dccf035ae5c06d41bc8f4284e7725c852503377e9a5"} Jul 10 19:03:59.934850 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:03:59.934809 2581 generic.go:356] "Generic (PLEG): container finished" podID="81f793e8-2ce7-46a9-8d5e-eff3b412a34f" containerID="762ecd1bc808999e86d714c3a3af46c6d7fe0bf2adad4af3cf5493b35a7899af" exitCode=0 Jul 10 19:03:59.935212 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:03:59.934864 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pn49" event={"ID":"81f793e8-2ce7-46a9-8d5e-eff3b412a34f","Type":"ContainerDied","Data":"762ecd1bc808999e86d714c3a3af46c6d7fe0bf2adad4af3cf5493b35a7899af"} Jul 10 19:04:02.949865 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:02.949825 2581 generic.go:356] "Generic (PLEG): container finished" podID="81f793e8-2ce7-46a9-8d5e-eff3b412a34f" containerID="724c37264d151673713220a2b4db6ea2ebaaffc8dcad4dc94ce8b8bed47bdc92" exitCode=0 Jul 10 19:04:02.950320 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:02.949913 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pn49" event={"ID":"81f793e8-2ce7-46a9-8d5e-eff3b412a34f","Type":"ContainerDied","Data":"724c37264d151673713220a2b4db6ea2ebaaffc8dcad4dc94ce8b8bed47bdc92"} Jul 10 19:04:08.978364 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:08.978270 2581 generic.go:356] "Generic (PLEG): container finished" podID="81f793e8-2ce7-46a9-8d5e-eff3b412a34f" containerID="3d45877bc99d156e2e5cda2529b43826e179d2bd69a2c3cd4fdeaa73f7106940" exitCode=0 Jul 10 19:04:08.978364 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:08.978341 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pn49" event={"ID":"81f793e8-2ce7-46a9-8d5e-eff3b412a34f","Type":"ContainerDied","Data":"3d45877bc99d156e2e5cda2529b43826e179d2bd69a2c3cd4fdeaa73f7106940"} Jul 10 19:04:10.110099 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:10.110074 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pn49" Jul 10 19:04:10.184786 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:10.184752 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/81f793e8-2ce7-46a9-8d5e-eff3b412a34f-bundle\") pod \"81f793e8-2ce7-46a9-8d5e-eff3b412a34f\" (UID: \"81f793e8-2ce7-46a9-8d5e-eff3b412a34f\") " Jul 10 19:04:10.184923 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:10.184796 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/81f793e8-2ce7-46a9-8d5e-eff3b412a34f-util\") pod \"81f793e8-2ce7-46a9-8d5e-eff3b412a34f\" (UID: \"81f793e8-2ce7-46a9-8d5e-eff3b412a34f\") " Jul 10 19:04:10.184923 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:10.184831 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kr7f8\" (UniqueName: \"kubernetes.io/projected/81f793e8-2ce7-46a9-8d5e-eff3b412a34f-kube-api-access-kr7f8\") pod \"81f793e8-2ce7-46a9-8d5e-eff3b412a34f\" (UID: \"81f793e8-2ce7-46a9-8d5e-eff3b412a34f\") " Jul 10 19:04:10.185400 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:10.185366 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81f793e8-2ce7-46a9-8d5e-eff3b412a34f-bundle" (OuterVolumeSpecName: "bundle") pod "81f793e8-2ce7-46a9-8d5e-eff3b412a34f" (UID: "81f793e8-2ce7-46a9-8d5e-eff3b412a34f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:04:10.186978 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:10.186955 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81f793e8-2ce7-46a9-8d5e-eff3b412a34f-kube-api-access-kr7f8" (OuterVolumeSpecName: "kube-api-access-kr7f8") pod "81f793e8-2ce7-46a9-8d5e-eff3b412a34f" (UID: "81f793e8-2ce7-46a9-8d5e-eff3b412a34f"). InnerVolumeSpecName "kube-api-access-kr7f8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:04:10.188892 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:10.188867 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81f793e8-2ce7-46a9-8d5e-eff3b412a34f-util" (OuterVolumeSpecName: "util") pod "81f793e8-2ce7-46a9-8d5e-eff3b412a34f" (UID: "81f793e8-2ce7-46a9-8d5e-eff3b412a34f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:04:10.285513 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:10.285479 2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/81f793e8-2ce7-46a9-8d5e-eff3b412a34f-bundle\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:04:10.285513 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:10.285512 2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/81f793e8-2ce7-46a9-8d5e-eff3b412a34f-util\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:04:10.285734 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:10.285527 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr7f8\" (UniqueName: \"kubernetes.io/projected/81f793e8-2ce7-46a9-8d5e-eff3b412a34f-kube-api-access-kr7f8\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:04:10.988645 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:10.988568 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pn49" Jul 10 19:04:10.988645 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:10.988572 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407pn49" event={"ID":"81f793e8-2ce7-46a9-8d5e-eff3b412a34f","Type":"ContainerDied","Data":"78561cc0cbfa688b6dcb0dccf035ae5c06d41bc8f4284e7725c852503377e9a5"} Jul 10 19:04:10.988645 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:10.988621 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78561cc0cbfa688b6dcb0dccf035ae5c06d41bc8f4284e7725c852503377e9a5" Jul 10 19:04:15.940547 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:15.940511 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t64rz"] Jul 10 19:04:15.940921 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:15.940816 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81f793e8-2ce7-46a9-8d5e-eff3b412a34f" containerName="pull" Jul 10 19:04:15.940921 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:15.940828 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="81f793e8-2ce7-46a9-8d5e-eff3b412a34f" containerName="pull" Jul 10 19:04:15.940921 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:15.940839 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81f793e8-2ce7-46a9-8d5e-eff3b412a34f" containerName="util" Jul 10 19:04:15.940921 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:15.940844 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="81f793e8-2ce7-46a9-8d5e-eff3b412a34f" containerName="util" Jul 10 19:04:15.940921 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:15.940851 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81f793e8-2ce7-46a9-8d5e-eff3b412a34f" containerName="extract" Jul 10 19:04:15.940921 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:15.940857 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="81f793e8-2ce7-46a9-8d5e-eff3b412a34f" containerName="extract" Jul 10 19:04:15.940921 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:15.940901 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="81f793e8-2ce7-46a9-8d5e-eff3b412a34f" containerName="extract" Jul 10 19:04:15.980311 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:15.980278 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t64rz"] Jul 10 19:04:15.980454 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:15.980371 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t64rz" Jul 10 19:04:15.983870 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:15.983850 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 10 19:04:15.983983 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:15.983921 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 10 19:04:15.987981 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:15.987963 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-74ksw\"" Jul 10 19:04:16.127902 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:16.127865 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4s9n\" (UniqueName: \"kubernetes.io/projected/94e7b83b-86e7-479d-8d8e-0dcccf18d7c8-kube-api-access-t4s9n\") pod \"cert-manager-operator-controller-manager-55c444bf86-t64rz\" (UID: \"94e7b83b-86e7-479d-8d8e-0dcccf18d7c8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t64rz" Jul 10 19:04:16.127902 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:16.127904 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/94e7b83b-86e7-479d-8d8e-0dcccf18d7c8-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-t64rz\" (UID: \"94e7b83b-86e7-479d-8d8e-0dcccf18d7c8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t64rz" Jul 10 19:04:16.229355 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:16.229258 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4s9n\" (UniqueName: \"kubernetes.io/projected/94e7b83b-86e7-479d-8d8e-0dcccf18d7c8-kube-api-access-t4s9n\") pod \"cert-manager-operator-controller-manager-55c444bf86-t64rz\" (UID: \"94e7b83b-86e7-479d-8d8e-0dcccf18d7c8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t64rz" Jul 10 19:04:16.229355 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:16.229303 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/94e7b83b-86e7-479d-8d8e-0dcccf18d7c8-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-t64rz\" (UID: \"94e7b83b-86e7-479d-8d8e-0dcccf18d7c8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t64rz" Jul 10 19:04:16.229717 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:16.229700 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/94e7b83b-86e7-479d-8d8e-0dcccf18d7c8-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-t64rz\" (UID: \"94e7b83b-86e7-479d-8d8e-0dcccf18d7c8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t64rz" Jul 10 19:04:16.237994 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:16.237963 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4s9n\" (UniqueName: \"kubernetes.io/projected/94e7b83b-86e7-479d-8d8e-0dcccf18d7c8-kube-api-access-t4s9n\") pod \"cert-manager-operator-controller-manager-55c444bf86-t64rz\" (UID: \"94e7b83b-86e7-479d-8d8e-0dcccf18d7c8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t64rz" Jul 10 19:04:16.289989 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:16.289952 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t64rz" Jul 10 19:04:16.458970 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:16.458941 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t64rz"] Jul 10 19:04:16.460808 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:04:16.460785 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94e7b83b_86e7_479d_8d8e_0dcccf18d7c8.slice/crio-67f0d18a20e5741ccabfd0d0ea3ef37c73aeac9f04c00850a86c0b9ea42ab8ac WatchSource:0}: Error finding container 67f0d18a20e5741ccabfd0d0ea3ef37c73aeac9f04c00850a86c0b9ea42ab8ac: Status 404 returned error can't find the container with id 67f0d18a20e5741ccabfd0d0ea3ef37c73aeac9f04c00850a86c0b9ea42ab8ac Jul 10 19:04:17.020238 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:17.020199 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t64rz" event={"ID":"94e7b83b-86e7-479d-8d8e-0dcccf18d7c8","Type":"ContainerStarted","Data":"67f0d18a20e5741ccabfd0d0ea3ef37c73aeac9f04c00850a86c0b9ea42ab8ac"} Jul 10 19:04:19.030199 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:19.030161 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t64rz" event={"ID":"94e7b83b-86e7-479d-8d8e-0dcccf18d7c8","Type":"ContainerStarted","Data":"55e1e2941adcd541f7ad913995e2c40c84e3e4bdea774fbb2276bdbaad7be6eb"} Jul 10 19:04:19.063125 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:19.063070 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t64rz" podStartSLOduration=1.873607619 podStartE2EDuration="4.063052553s" totalImagesPullingTime="2.189444934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:04:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:04:19.06110307 +0000 UTC m=+343.105245603" watchObservedRunningTime="2026-07-10 19:04:19.063052553 +0000 UTC m=+343.107195053" Jul 10 19:04:19.688620 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:19.688585 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9mwl8"] Jul 10 19:04:19.711142 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:19.711110 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9mwl8" Jul 10 19:04:19.713549 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:19.713522 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9mwl8"] Jul 10 19:04:19.714375 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:19.714351 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-b9hpz\"" Jul 10 19:04:19.714559 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:19.714544 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 10 19:04:19.716498 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:19.716473 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 10 19:04:19.859732 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:19.859670 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d717ce93-f827-41b4-8706-b10d5d921244-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9mwl8\" (UID: \"d717ce93-f827-41b4-8706-b10d5d921244\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9mwl8" Jul 10 19:04:19.859732 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:19.859733 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d717ce93-f827-41b4-8706-b10d5d921244-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9mwl8\" (UID: \"d717ce93-f827-41b4-8706-b10d5d921244\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9mwl8" Jul 10 19:04:19.859948 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:19.859778 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5csdn\" (UniqueName: \"kubernetes.io/projected/d717ce93-f827-41b4-8706-b10d5d921244-kube-api-access-5csdn\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9mwl8\" (UID: \"d717ce93-f827-41b4-8706-b10d5d921244\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9mwl8" Jul 10 19:04:19.961258 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:19.961163 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d717ce93-f827-41b4-8706-b10d5d921244-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9mwl8\" (UID: \"d717ce93-f827-41b4-8706-b10d5d921244\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9mwl8" Jul 10 19:04:19.961258 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:19.961199 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5csdn\" (UniqueName: \"kubernetes.io/projected/d717ce93-f827-41b4-8706-b10d5d921244-kube-api-access-5csdn\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9mwl8\" (UID: \"d717ce93-f827-41b4-8706-b10d5d921244\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9mwl8" Jul 10 19:04:19.961437 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:19.961325 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d717ce93-f827-41b4-8706-b10d5d921244-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9mwl8\" (UID: \"d717ce93-f827-41b4-8706-b10d5d921244\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9mwl8" Jul 10 19:04:19.961654 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:19.961633 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d717ce93-f827-41b4-8706-b10d5d921244-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9mwl8\" (UID: \"d717ce93-f827-41b4-8706-b10d5d921244\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9mwl8" Jul 10 19:04:19.961718 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:19.961645 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d717ce93-f827-41b4-8706-b10d5d921244-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9mwl8\" (UID: \"d717ce93-f827-41b4-8706-b10d5d921244\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9mwl8" Jul 10 19:04:20.018613 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:20.018580 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5csdn\" (UniqueName: \"kubernetes.io/projected/d717ce93-f827-41b4-8706-b10d5d921244-kube-api-access-5csdn\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9mwl8\" (UID: \"d717ce93-f827-41b4-8706-b10d5d921244\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9mwl8" Jul 10 19:04:20.022372 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:20.022354 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9mwl8" Jul 10 19:04:20.169855 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:20.169829 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9mwl8"] Jul 10 19:04:20.172009 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:04:20.171978 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd717ce93_f827_41b4_8706_b10d5d921244.slice/crio-1a57c882bd491d1ed88be144f02f2de0c9ae2c61c5f1dfbae758cfb06b8549b2 WatchSource:0}: Error finding container 1a57c882bd491d1ed88be144f02f2de0c9ae2c61c5f1dfbae758cfb06b8549b2: Status 404 returned error can't find the container with id 1a57c882bd491d1ed88be144f02f2de0c9ae2c61c5f1dfbae758cfb06b8549b2 Jul 10 19:04:21.039005 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:21.038975 2581 generic.go:356] "Generic (PLEG): container finished" podID="d717ce93-f827-41b4-8706-b10d5d921244" containerID="3e67b713be0c7673a0bdf918ee35cfab0a84c68d83e7e2e4208c6c9dedd4956a" exitCode=0 Jul 10 19:04:21.039158 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:21.039013 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9mwl8" event={"ID":"d717ce93-f827-41b4-8706-b10d5d921244","Type":"ContainerDied","Data":"3e67b713be0c7673a0bdf918ee35cfab0a84c68d83e7e2e4208c6c9dedd4956a"} Jul 10 19:04:21.039158 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:21.039047 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9mwl8" event={"ID":"d717ce93-f827-41b4-8706-b10d5d921244","Type":"ContainerStarted","Data":"1a57c882bd491d1ed88be144f02f2de0c9ae2c61c5f1dfbae758cfb06b8549b2"} Jul 10 19:04:23.049924 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:23.049880 2581 generic.go:356] "Generic (PLEG): container finished" podID="d717ce93-f827-41b4-8706-b10d5d921244" containerID="5375727a08e357d97b2786a12e0f9eec68ecb3c3f0d912b2c15b0faede951a79" exitCode=0 Jul 10 19:04:23.050301 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:23.049945 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9mwl8" event={"ID":"d717ce93-f827-41b4-8706-b10d5d921244","Type":"ContainerDied","Data":"5375727a08e357d97b2786a12e0f9eec68ecb3c3f0d912b2c15b0faede951a79"} Jul 10 19:04:24.056288 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:24.056253 2581 generic.go:356] "Generic (PLEG): container finished" podID="d717ce93-f827-41b4-8706-b10d5d921244" containerID="4800eaf018543803e365708d489f4acc93dbae2640e9fd9cbab4fc4988ed3d14" exitCode=0 Jul 10 19:04:24.056816 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:24.056342 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9mwl8" event={"ID":"d717ce93-f827-41b4-8706-b10d5d921244","Type":"ContainerDied","Data":"4800eaf018543803e365708d489f4acc93dbae2640e9fd9cbab4fc4988ed3d14"} Jul 10 19:04:25.183083 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:25.183059 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9mwl8" Jul 10 19:04:25.302440 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:25.302398 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d717ce93-f827-41b4-8706-b10d5d921244-util\") pod \"d717ce93-f827-41b4-8706-b10d5d921244\" (UID: \"d717ce93-f827-41b4-8706-b10d5d921244\") " Jul 10 19:04:25.302634 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:25.302473 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5csdn\" (UniqueName: \"kubernetes.io/projected/d717ce93-f827-41b4-8706-b10d5d921244-kube-api-access-5csdn\") pod \"d717ce93-f827-41b4-8706-b10d5d921244\" (UID: \"d717ce93-f827-41b4-8706-b10d5d921244\") " Jul 10 19:04:25.302634 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:25.302530 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d717ce93-f827-41b4-8706-b10d5d921244-bundle\") pod \"d717ce93-f827-41b4-8706-b10d5d921244\" (UID: \"d717ce93-f827-41b4-8706-b10d5d921244\") " Jul 10 19:04:25.302993 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:25.302959 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d717ce93-f827-41b4-8706-b10d5d921244-bundle" (OuterVolumeSpecName: "bundle") pod "d717ce93-f827-41b4-8706-b10d5d921244" (UID: "d717ce93-f827-41b4-8706-b10d5d921244"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:04:25.304636 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:25.304611 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d717ce93-f827-41b4-8706-b10d5d921244-kube-api-access-5csdn" (OuterVolumeSpecName: "kube-api-access-5csdn") pod "d717ce93-f827-41b4-8706-b10d5d921244" (UID: "d717ce93-f827-41b4-8706-b10d5d921244"). InnerVolumeSpecName "kube-api-access-5csdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:04:25.308506 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:25.308481 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d717ce93-f827-41b4-8706-b10d5d921244-util" (OuterVolumeSpecName: "util") pod "d717ce93-f827-41b4-8706-b10d5d921244" (UID: "d717ce93-f827-41b4-8706-b10d5d921244"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:04:25.403464 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:25.403366 2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d717ce93-f827-41b4-8706-b10d5d921244-bundle\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:04:25.403464 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:25.403406 2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d717ce93-f827-41b4-8706-b10d5d921244-util\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:04:25.403464 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:25.403415 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5csdn\" (UniqueName: \"kubernetes.io/projected/d717ce93-f827-41b4-8706-b10d5d921244-kube-api-access-5csdn\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:04:26.066174 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:26.066146 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9mwl8" Jul 10 19:04:26.066343 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:26.066145 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9mwl8" event={"ID":"d717ce93-f827-41b4-8706-b10d5d921244","Type":"ContainerDied","Data":"1a57c882bd491d1ed88be144f02f2de0c9ae2c61c5f1dfbae758cfb06b8549b2"} Jul 10 19:04:26.066343 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:26.066273 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a57c882bd491d1ed88be144f02f2de0c9ae2c61c5f1dfbae758cfb06b8549b2" Jul 10 19:04:26.881410 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:26.881374 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-fwv44"] Jul 10 19:04:26.881809 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:26.881706 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d717ce93-f827-41b4-8706-b10d5d921244" containerName="extract" Jul 10 19:04:26.881809 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:26.881718 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d717ce93-f827-41b4-8706-b10d5d921244" containerName="extract" Jul 10 19:04:26.881809 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:26.881728 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d717ce93-f827-41b4-8706-b10d5d921244" containerName="util" Jul 10 19:04:26.881809 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:26.881734 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d717ce93-f827-41b4-8706-b10d5d921244" containerName="util" Jul 10 19:04:26.881809 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:26.881741 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d717ce93-f827-41b4-8706-b10d5d921244" containerName="pull" Jul 10 19:04:26.881809 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:26.881747 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d717ce93-f827-41b4-8706-b10d5d921244" containerName="pull" Jul 10 19:04:26.881809 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:26.881791 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d717ce93-f827-41b4-8706-b10d5d921244" containerName="extract" Jul 10 19:04:26.883654 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:26.883635 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-fwv44" Jul 10 19:04:26.886879 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:26.886856 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 10 19:04:26.887024 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:26.886894 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 10 19:04:26.887464 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:26.887448 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-qndw7\"" Jul 10 19:04:26.897909 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:26.897753 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-fwv44"] Jul 10 19:04:27.014678 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:27.014632 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4jnm\" (UniqueName: \"kubernetes.io/projected/7788f7d1-1fbe-4ad0-b697-1d0ad4b12e70-kube-api-access-q4jnm\") pod \"cert-manager-webhook-6486ffd997-fwv44\" (UID: \"7788f7d1-1fbe-4ad0-b697-1d0ad4b12e70\") " pod="cert-manager/cert-manager-webhook-6486ffd997-fwv44" Jul 10 19:04:27.014862 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:27.014703 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7788f7d1-1fbe-4ad0-b697-1d0ad4b12e70-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-fwv44\" (UID: \"7788f7d1-1fbe-4ad0-b697-1d0ad4b12e70\") " pod="cert-manager/cert-manager-webhook-6486ffd997-fwv44" Jul 10 19:04:27.116116 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:27.116078 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4jnm\" (UniqueName: \"kubernetes.io/projected/7788f7d1-1fbe-4ad0-b697-1d0ad4b12e70-kube-api-access-q4jnm\") pod \"cert-manager-webhook-6486ffd997-fwv44\" (UID: \"7788f7d1-1fbe-4ad0-b697-1d0ad4b12e70\") " pod="cert-manager/cert-manager-webhook-6486ffd997-fwv44" Jul 10 19:04:27.116306 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:27.116136 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7788f7d1-1fbe-4ad0-b697-1d0ad4b12e70-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-fwv44\" (UID: \"7788f7d1-1fbe-4ad0-b697-1d0ad4b12e70\") " pod="cert-manager/cert-manager-webhook-6486ffd997-fwv44" Jul 10 19:04:27.125787 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:27.125762 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7788f7d1-1fbe-4ad0-b697-1d0ad4b12e70-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-fwv44\" (UID: \"7788f7d1-1fbe-4ad0-b697-1d0ad4b12e70\") " pod="cert-manager/cert-manager-webhook-6486ffd997-fwv44" Jul 10 19:04:27.126064 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:27.126044 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4jnm\" (UniqueName: \"kubernetes.io/projected/7788f7d1-1fbe-4ad0-b697-1d0ad4b12e70-kube-api-access-q4jnm\") pod \"cert-manager-webhook-6486ffd997-fwv44\" (UID: \"7788f7d1-1fbe-4ad0-b697-1d0ad4b12e70\") " pod="cert-manager/cert-manager-webhook-6486ffd997-fwv44" Jul 10 19:04:27.203802 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:27.203679 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-fwv44" Jul 10 19:04:27.351779 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:27.351756 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-fwv44"] Jul 10 19:04:27.353915 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:04:27.353885 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7788f7d1_1fbe_4ad0_b697_1d0ad4b12e70.slice/crio-62f8b16fb432cc71cd50058cac396ceecfbd75b02a0d191a5e04aa22dbbd715b WatchSource:0}: Error finding container 62f8b16fb432cc71cd50058cac396ceecfbd75b02a0d191a5e04aa22dbbd715b: Status 404 returned error can't find the container with id 62f8b16fb432cc71cd50058cac396ceecfbd75b02a0d191a5e04aa22dbbd715b Jul 10 19:04:28.076263 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:28.076222 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-fwv44" event={"ID":"7788f7d1-1fbe-4ad0-b697-1d0ad4b12e70","Type":"ContainerStarted","Data":"62f8b16fb432cc71cd50058cac396ceecfbd75b02a0d191a5e04aa22dbbd715b"} Jul 10 19:04:30.085571 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:30.085472 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-fwv44" event={"ID":"7788f7d1-1fbe-4ad0-b697-1d0ad4b12e70","Type":"ContainerStarted","Data":"4b0cd661bdb8eb185fb3109a4a96db266e385c1ae26f2cfcfafbbf1b7ff74d9d"} Jul 10 19:04:30.085571 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:30.085542 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-fwv44" Jul 10 19:04:30.104941 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:30.104852 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-fwv44" podStartSLOduration=1.680252677 podStartE2EDuration="4.104833314s" totalImagesPullingTime="2.424580637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:04:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:04:30.103743996 +0000 UTC m=+354.147886498" watchObservedRunningTime="2026-07-10 19:04:30.104833314 +0000 UTC m=+354.148975817" Jul 10 19:04:36.092324 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:36.092289 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-fwv44" Jul 10 19:04:39.731596 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:39.731557 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm8vws"] Jul 10 19:04:39.734103 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:39.734087 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm8vws" Jul 10 19:04:39.736333 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:39.736308 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-b9hpz\"" Jul 10 19:04:39.736429 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:39.736394 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 10 19:04:39.737237 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:39.737218 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 10 19:04:39.743301 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:39.743277 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm8vws"] Jul 10 19:04:39.817573 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:39.817533 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/add2296d-2354-484d-872a-32aa133461cb-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm8vws\" (UID: \"add2296d-2354-484d-872a-32aa133461cb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm8vws" Jul 10 19:04:39.817573 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:39.817569 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/add2296d-2354-484d-872a-32aa133461cb-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm8vws\" (UID: \"add2296d-2354-484d-872a-32aa133461cb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm8vws" Jul 10 19:04:39.817813 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:39.817604 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj7kw\" (UniqueName: \"kubernetes.io/projected/add2296d-2354-484d-872a-32aa133461cb-kube-api-access-qj7kw\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm8vws\" (UID: \"add2296d-2354-484d-872a-32aa133461cb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm8vws" Jul 10 19:04:39.918992 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:39.918953 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/add2296d-2354-484d-872a-32aa133461cb-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm8vws\" (UID: \"add2296d-2354-484d-872a-32aa133461cb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm8vws" Jul 10 19:04:39.918992 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:39.918990 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/add2296d-2354-484d-872a-32aa133461cb-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm8vws\" (UID: \"add2296d-2354-484d-872a-32aa133461cb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm8vws" Jul 10 19:04:39.919194 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:39.919020 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj7kw\" (UniqueName: \"kubernetes.io/projected/add2296d-2354-484d-872a-32aa133461cb-kube-api-access-qj7kw\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm8vws\" (UID: \"add2296d-2354-484d-872a-32aa133461cb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm8vws" Jul 10 19:04:39.919427 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:39.919406 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/add2296d-2354-484d-872a-32aa133461cb-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm8vws\" (UID: \"add2296d-2354-484d-872a-32aa133461cb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm8vws" Jul 10 19:04:39.919427 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:39.919417 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/add2296d-2354-484d-872a-32aa133461cb-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm8vws\" (UID: \"add2296d-2354-484d-872a-32aa133461cb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm8vws" Jul 10 19:04:39.927717 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:39.927672 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj7kw\" (UniqueName: \"kubernetes.io/projected/add2296d-2354-484d-872a-32aa133461cb-kube-api-access-qj7kw\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm8vws\" (UID: \"add2296d-2354-484d-872a-32aa133461cb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm8vws" Jul 10 19:04:40.043819 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:40.043778 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm8vws" Jul 10 19:04:40.179010 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:40.178983 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm8vws"] Jul 10 19:04:40.181631 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:04:40.181601 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadd2296d_2354_484d_872a_32aa133461cb.slice/crio-3dc9c911d38db0a758aa80a95bd076dc8bd3ff35d7fde1752d8010ce59c0d8ae WatchSource:0}: Error finding container 3dc9c911d38db0a758aa80a95bd076dc8bd3ff35d7fde1752d8010ce59c0d8ae: Status 404 returned error can't find the container with id 3dc9c911d38db0a758aa80a95bd076dc8bd3ff35d7fde1752d8010ce59c0d8ae Jul 10 19:04:41.134878 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:41.134844 2581 generic.go:356] "Generic (PLEG): container finished" podID="add2296d-2354-484d-872a-32aa133461cb" containerID="456eddca2e51226d9e8d741f6c41bcc3b4d7ef395d06e362f3b955d26be609a8" exitCode=0 Jul 10 19:04:41.135260 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:41.134928 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm8vws" event={"ID":"add2296d-2354-484d-872a-32aa133461cb","Type":"ContainerDied","Data":"456eddca2e51226d9e8d741f6c41bcc3b4d7ef395d06e362f3b955d26be609a8"} Jul 10 19:04:41.135260 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:41.134979 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm8vws" event={"ID":"add2296d-2354-484d-872a-32aa133461cb","Type":"ContainerStarted","Data":"3dc9c911d38db0a758aa80a95bd076dc8bd3ff35d7fde1752d8010ce59c0d8ae"} Jul 10 19:04:42.141165 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:42.141119 2581 generic.go:356] "Generic (PLEG): container finished" podID="add2296d-2354-484d-872a-32aa133461cb" containerID="c8bf65804b42d55747233ec5b3cf34226ad34cc0c6dedb5317990e03b1f60ba3" exitCode=0 Jul 10 19:04:42.141631 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:42.141177 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm8vws" event={"ID":"add2296d-2354-484d-872a-32aa133461cb","Type":"ContainerDied","Data":"c8bf65804b42d55747233ec5b3cf34226ad34cc0c6dedb5317990e03b1f60ba3"} Jul 10 19:04:43.146849 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:43.146814 2581 generic.go:356] "Generic (PLEG): container finished" podID="add2296d-2354-484d-872a-32aa133461cb" containerID="b685a131eb7f9ee5b8bc00747711c4e10a8861e9698aa316949074f56adcf8b0" exitCode=0 Jul 10 19:04:43.147276 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:43.146882 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm8vws" event={"ID":"add2296d-2354-484d-872a-32aa133461cb","Type":"ContainerDied","Data":"b685a131eb7f9ee5b8bc00747711c4e10a8861e9698aa316949074f56adcf8b0"} Jul 10 19:04:44.277212 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:44.277185 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm8vws" Jul 10 19:04:44.350981 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:44.350945 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/add2296d-2354-484d-872a-32aa133461cb-bundle\") pod \"add2296d-2354-484d-872a-32aa133461cb\" (UID: \"add2296d-2354-484d-872a-32aa133461cb\") " Jul 10 19:04:44.350981 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:44.350987 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qj7kw\" (UniqueName: \"kubernetes.io/projected/add2296d-2354-484d-872a-32aa133461cb-kube-api-access-qj7kw\") pod \"add2296d-2354-484d-872a-32aa133461cb\" (UID: \"add2296d-2354-484d-872a-32aa133461cb\") " Jul 10 19:04:44.351220 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:44.351023 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/add2296d-2354-484d-872a-32aa133461cb-util\") pod \"add2296d-2354-484d-872a-32aa133461cb\" (UID: \"add2296d-2354-484d-872a-32aa133461cb\") " Jul 10 19:04:44.351695 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:44.351662 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/add2296d-2354-484d-872a-32aa133461cb-bundle" (OuterVolumeSpecName: "bundle") pod "add2296d-2354-484d-872a-32aa133461cb" (UID: "add2296d-2354-484d-872a-32aa133461cb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:04:44.353108 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:44.353082 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/add2296d-2354-484d-872a-32aa133461cb-kube-api-access-qj7kw" (OuterVolumeSpecName: "kube-api-access-qj7kw") pod "add2296d-2354-484d-872a-32aa133461cb" (UID: "add2296d-2354-484d-872a-32aa133461cb"). InnerVolumeSpecName "kube-api-access-qj7kw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:04:44.358423 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:44.358381 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/add2296d-2354-484d-872a-32aa133461cb-util" (OuterVolumeSpecName: "util") pod "add2296d-2354-484d-872a-32aa133461cb" (UID: "add2296d-2354-484d-872a-32aa133461cb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:04:44.452337 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:44.452257 2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/add2296d-2354-484d-872a-32aa133461cb-bundle\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:04:44.452337 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:44.452285 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj7kw\" (UniqueName: \"kubernetes.io/projected/add2296d-2354-484d-872a-32aa133461cb-kube-api-access-qj7kw\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:04:44.452337 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:44.452295 2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/add2296d-2354-484d-872a-32aa133461cb-util\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:04:45.157174 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:45.157141 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm8vws" event={"ID":"add2296d-2354-484d-872a-32aa133461cb","Type":"ContainerDied","Data":"3dc9c911d38db0a758aa80a95bd076dc8bd3ff35d7fde1752d8010ce59c0d8ae"} Jul 10 19:04:45.157174 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:45.157172 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm8vws" Jul 10 19:04:45.157380 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:45.157194 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3dc9c911d38db0a758aa80a95bd076dc8bd3ff35d7fde1752d8010ce59c0d8ae" Jul 10 19:04:57.194587 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:57.194545 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-wwbcb"] Jul 10 19:04:57.195175 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:57.194977 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="add2296d-2354-484d-872a-32aa133461cb" containerName="pull" Jul 10 19:04:57.195175 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:57.194996 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="add2296d-2354-484d-872a-32aa133461cb" containerName="pull" Jul 10 19:04:57.195175 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:57.195021 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="add2296d-2354-484d-872a-32aa133461cb" containerName="util" Jul 10 19:04:57.195175 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:57.195029 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="add2296d-2354-484d-872a-32aa133461cb" containerName="util" Jul 10 19:04:57.195175 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:57.195041 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="add2296d-2354-484d-872a-32aa133461cb" containerName="extract" Jul 10 19:04:57.195175 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:57.195050 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="add2296d-2354-484d-872a-32aa133461cb" containerName="extract" Jul 10 19:04:57.195175 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:57.195141 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="add2296d-2354-484d-872a-32aa133461cb" containerName="extract" Jul 10 19:04:57.201748 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:57.201725 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-wwbcb" Jul 10 19:04:57.205637 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:57.205612 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 10 19:04:57.206953 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:57.206913 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 10 19:04:57.207126 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:57.207100 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 10 19:04:57.211939 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:57.211920 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-kshvj\"" Jul 10 19:04:57.212819 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:57.212805 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 10 19:04:57.220106 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:57.220082 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-wwbcb"] Jul 10 19:04:57.260982 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:57.260957 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3c64a97c-e4f5-49d7-ad20-18a4ec847644-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7cfcf59dd-wwbcb\" (UID: \"3c64a97c-e4f5-49d7-ad20-18a4ec847644\") " pod="opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-wwbcb" Jul 10 19:04:57.261152 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:57.260988 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9ksg\" (UniqueName: \"kubernetes.io/projected/3c64a97c-e4f5-49d7-ad20-18a4ec847644-kube-api-access-c9ksg\") pod \"opendatahub-operator-controller-manager-7cfcf59dd-wwbcb\" (UID: \"3c64a97c-e4f5-49d7-ad20-18a4ec847644\") " pod="opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-wwbcb" Jul 10 19:04:57.261152 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:57.261017 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c64a97c-e4f5-49d7-ad20-18a4ec847644-tmp\") pod \"opendatahub-operator-controller-manager-7cfcf59dd-wwbcb\" (UID: \"3c64a97c-e4f5-49d7-ad20-18a4ec847644\") " pod="opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-wwbcb" Jul 10 19:04:57.261271 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:57.261148 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3c64a97c-e4f5-49d7-ad20-18a4ec847644-webhook-cert\") pod \"opendatahub-operator-controller-manager-7cfcf59dd-wwbcb\" (UID: \"3c64a97c-e4f5-49d7-ad20-18a4ec847644\") " pod="opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-wwbcb" Jul 10 19:04:57.261271 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:57.261207 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3c64a97c-e4f5-49d7-ad20-18a4ec847644-manifests\") pod \"opendatahub-operator-controller-manager-7cfcf59dd-wwbcb\" (UID: \"3c64a97c-e4f5-49d7-ad20-18a4ec847644\") " pod="opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-wwbcb" Jul 10 19:04:57.361594 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:57.361561 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9ksg\" (UniqueName: \"kubernetes.io/projected/3c64a97c-e4f5-49d7-ad20-18a4ec847644-kube-api-access-c9ksg\") pod \"opendatahub-operator-controller-manager-7cfcf59dd-wwbcb\" (UID: \"3c64a97c-e4f5-49d7-ad20-18a4ec847644\") " pod="opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-wwbcb" Jul 10 19:04:57.361808 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:57.361611 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c64a97c-e4f5-49d7-ad20-18a4ec847644-tmp\") pod \"opendatahub-operator-controller-manager-7cfcf59dd-wwbcb\" (UID: \"3c64a97c-e4f5-49d7-ad20-18a4ec847644\") " pod="opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-wwbcb" Jul 10 19:04:57.361808 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:57.361653 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3c64a97c-e4f5-49d7-ad20-18a4ec847644-webhook-cert\") pod \"opendatahub-operator-controller-manager-7cfcf59dd-wwbcb\" (UID: \"3c64a97c-e4f5-49d7-ad20-18a4ec847644\") " pod="opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-wwbcb" Jul 10 19:04:57.361808 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:57.361679 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3c64a97c-e4f5-49d7-ad20-18a4ec847644-manifests\") pod \"opendatahub-operator-controller-manager-7cfcf59dd-wwbcb\" (UID: \"3c64a97c-e4f5-49d7-ad20-18a4ec847644\") " pod="opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-wwbcb" Jul 10 19:04:57.361808 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:57.361724 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3c64a97c-e4f5-49d7-ad20-18a4ec847644-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7cfcf59dd-wwbcb\" (UID: \"3c64a97c-e4f5-49d7-ad20-18a4ec847644\") " pod="opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-wwbcb" Jul 10 19:04:57.362194 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:57.362165 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3c64a97c-e4f5-49d7-ad20-18a4ec847644-manifests\") pod \"opendatahub-operator-controller-manager-7cfcf59dd-wwbcb\" (UID: \"3c64a97c-e4f5-49d7-ad20-18a4ec847644\") " pod="opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-wwbcb" Jul 10 19:04:57.364058 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:57.364027 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c64a97c-e4f5-49d7-ad20-18a4ec847644-tmp\") pod \"opendatahub-operator-controller-manager-7cfcf59dd-wwbcb\" (UID: \"3c64a97c-e4f5-49d7-ad20-18a4ec847644\") " pod="opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-wwbcb" Jul 10 19:04:57.364181 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:57.364163 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3c64a97c-e4f5-49d7-ad20-18a4ec847644-webhook-cert\") pod \"opendatahub-operator-controller-manager-7cfcf59dd-wwbcb\" (UID: \"3c64a97c-e4f5-49d7-ad20-18a4ec847644\") " pod="opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-wwbcb" Jul 10 19:04:57.364258 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:57.364241 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3c64a97c-e4f5-49d7-ad20-18a4ec847644-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7cfcf59dd-wwbcb\" (UID: \"3c64a97c-e4f5-49d7-ad20-18a4ec847644\") " pod="opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-wwbcb" Jul 10 19:04:57.371111 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:57.371088 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9ksg\" (UniqueName: \"kubernetes.io/projected/3c64a97c-e4f5-49d7-ad20-18a4ec847644-kube-api-access-c9ksg\") pod \"opendatahub-operator-controller-manager-7cfcf59dd-wwbcb\" (UID: \"3c64a97c-e4f5-49d7-ad20-18a4ec847644\") " pod="opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-wwbcb" Jul 10 19:04:57.511544 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:57.511513 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-wwbcb" Jul 10 19:04:57.648163 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:57.648137 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-wwbcb"] Jul 10 19:04:57.650317 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:04:57.650277 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c64a97c_e4f5_49d7_ad20_18a4ec847644.slice/crio-df9eaa4e836c809ca8d61cc835b8cd6989e69a7b59cf9a1ff8c82fe690f04825 WatchSource:0}: Error finding container df9eaa4e836c809ca8d61cc835b8cd6989e69a7b59cf9a1ff8c82fe690f04825: Status 404 returned error can't find the container with id df9eaa4e836c809ca8d61cc835b8cd6989e69a7b59cf9a1ff8c82fe690f04825 Jul 10 19:04:58.212079 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:04:58.211958 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-wwbcb" event={"ID":"3c64a97c-e4f5-49d7-ad20-18a4ec847644","Type":"ContainerStarted","Data":"df9eaa4e836c809ca8d61cc835b8cd6989e69a7b59cf9a1ff8c82fe690f04825"} Jul 10 19:05:01.229029 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:01.228991 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-wwbcb" event={"ID":"3c64a97c-e4f5-49d7-ad20-18a4ec847644","Type":"ContainerStarted","Data":"c32f39399a1c9d1ac3de915e94ac56ce0687cde6b528dd25b24ac6a4bbd4dab7"} Jul 10 19:05:02.234669 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:02.234632 2581 generic.go:356] "Generic (PLEG): container finished" podID="3c64a97c-e4f5-49d7-ad20-18a4ec847644" containerID="c32f39399a1c9d1ac3de915e94ac56ce0687cde6b528dd25b24ac6a4bbd4dab7" exitCode=0 Jul 10 19:05:02.235092 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:02.234713 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-wwbcb" event={"ID":"3c64a97c-e4f5-49d7-ad20-18a4ec847644","Type":"ContainerDied","Data":"c32f39399a1c9d1ac3de915e94ac56ce0687cde6b528dd25b24ac6a4bbd4dab7"} Jul 10 19:05:03.241188 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:03.241150 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-wwbcb" event={"ID":"3c64a97c-e4f5-49d7-ad20-18a4ec847644","Type":"ContainerStarted","Data":"5f1e37352f2118e1ad21d3200fe3c693e06f6c5b39ab3a8dad9c1cbc5ffdbbeb"} Jul 10 19:05:03.241593 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:03.241325 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-wwbcb" Jul 10 19:05:03.268756 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:03.268679 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-wwbcb" podStartSLOduration=1.996754895 podStartE2EDuration="6.268658667s" totalImagesPullingTime="3.198255176s" totalInitContainerRuntime="1.073648596s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:04:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:05:03.26641188 +0000 UTC m=+387.310554381" watchObservedRunningTime="2026-07-10 19:05:03.268658667 +0000 UTC m=+387.312801172" Jul 10 19:05:14.250176 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:14.250143 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7cfcf59dd-wwbcb" Jul 10 19:05:19.715757 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:19.715720 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-9g4cr"] Jul 10 19:05:19.724240 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:19.724217 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-9g4cr" Jul 10 19:05:19.731006 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:19.730935 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-85snc\"" Jul 10 19:05:19.733032 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:19.733011 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 10 19:05:19.834352 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:19.834312 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-9g4cr"] Jul 10 19:05:19.850645 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:19.850613 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/29957d41-1783-4280-bee2-75c4529ab426-cert\") pod \"odh-model-controller-7998c67dc4-9g4cr\" (UID: \"29957d41-1783-4280-bee2-75c4529ab426\") " pod="opendatahub/odh-model-controller-7998c67dc4-9g4cr" Jul 10 19:05:19.850820 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:19.850654 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmq7t\" (UniqueName: \"kubernetes.io/projected/29957d41-1783-4280-bee2-75c4529ab426-kube-api-access-hmq7t\") pod \"odh-model-controller-7998c67dc4-9g4cr\" (UID: \"29957d41-1783-4280-bee2-75c4529ab426\") " pod="opendatahub/odh-model-controller-7998c67dc4-9g4cr" Jul 10 19:05:19.951739 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:19.951680 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmq7t\" (UniqueName: \"kubernetes.io/projected/29957d41-1783-4280-bee2-75c4529ab426-kube-api-access-hmq7t\") pod \"odh-model-controller-7998c67dc4-9g4cr\" (UID: \"29957d41-1783-4280-bee2-75c4529ab426\") " pod="opendatahub/odh-model-controller-7998c67dc4-9g4cr" Jul 10 19:05:19.951943 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:19.951795 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/29957d41-1783-4280-bee2-75c4529ab426-cert\") pod \"odh-model-controller-7998c67dc4-9g4cr\" (UID: \"29957d41-1783-4280-bee2-75c4529ab426\") " pod="opendatahub/odh-model-controller-7998c67dc4-9g4cr" Jul 10 19:05:19.951943 ip-10-0-135-67 kubenswrapper[2581]: E0710 19:05:19.951887 2581 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 10 19:05:19.952064 ip-10-0-135-67 kubenswrapper[2581]: E0710 19:05:19.951975 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29957d41-1783-4280-bee2-75c4529ab426-cert podName:29957d41-1783-4280-bee2-75c4529ab426 nodeName:}" failed. No retries permitted until 2026-07-10 19:05:20.451955151 +0000 UTC m=+404.496097644 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/29957d41-1783-4280-bee2-75c4529ab426-cert") pod "odh-model-controller-7998c67dc4-9g4cr" (UID: "29957d41-1783-4280-bee2-75c4529ab426") : secret "odh-model-controller-webhook-cert" not found Jul 10 19:05:20.104185 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:20.104147 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmq7t\" (UniqueName: \"kubernetes.io/projected/29957d41-1783-4280-bee2-75c4529ab426-kube-api-access-hmq7t\") pod \"odh-model-controller-7998c67dc4-9g4cr\" (UID: \"29957d41-1783-4280-bee2-75c4529ab426\") " pod="opendatahub/odh-model-controller-7998c67dc4-9g4cr" Jul 10 19:05:20.349089 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:20.349051 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-697457d895-hj6vs"] Jul 10 19:05:20.353022 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:20.353001 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-697457d895-hj6vs" Jul 10 19:05:20.378036 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:20.378008 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 10 19:05:20.378276 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:20.378261 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-sdgrg\"" Jul 10 19:05:20.407162 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:20.407132 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 10 19:05:20.455908 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:20.455862 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/29957d41-1783-4280-bee2-75c4529ab426-cert\") pod \"odh-model-controller-7998c67dc4-9g4cr\" (UID: \"29957d41-1783-4280-bee2-75c4529ab426\") " pod="opendatahub/odh-model-controller-7998c67dc4-9g4cr" Jul 10 19:05:20.456096 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:20.455933 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d275207d-476a-4ce4-9ce2-da76a36e658d-tls-certs\") pod \"kube-auth-proxy-697457d895-hj6vs\" (UID: \"d275207d-476a-4ce4-9ce2-da76a36e658d\") " pod="openshift-ingress/kube-auth-proxy-697457d895-hj6vs" Jul 10 19:05:20.456096 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:20.455994 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lfkw\" (UniqueName: \"kubernetes.io/projected/d275207d-476a-4ce4-9ce2-da76a36e658d-kube-api-access-5lfkw\") pod \"kube-auth-proxy-697457d895-hj6vs\" (UID: \"d275207d-476a-4ce4-9ce2-da76a36e658d\") " pod="openshift-ingress/kube-auth-proxy-697457d895-hj6vs" Jul 10 19:05:20.456096 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:20.456019 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d275207d-476a-4ce4-9ce2-da76a36e658d-tmp\") pod \"kube-auth-proxy-697457d895-hj6vs\" (UID: \"d275207d-476a-4ce4-9ce2-da76a36e658d\") " pod="openshift-ingress/kube-auth-proxy-697457d895-hj6vs" Jul 10 19:05:20.458369 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:20.458344 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/29957d41-1783-4280-bee2-75c4529ab426-cert\") pod \"odh-model-controller-7998c67dc4-9g4cr\" (UID: \"29957d41-1783-4280-bee2-75c4529ab426\") " pod="opendatahub/odh-model-controller-7998c67dc4-9g4cr" Jul 10 19:05:20.556644 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:20.556603 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lfkw\" (UniqueName: \"kubernetes.io/projected/d275207d-476a-4ce4-9ce2-da76a36e658d-kube-api-access-5lfkw\") pod \"kube-auth-proxy-697457d895-hj6vs\" (UID: \"d275207d-476a-4ce4-9ce2-da76a36e658d\") " pod="openshift-ingress/kube-auth-proxy-697457d895-hj6vs" Jul 10 19:05:20.556856 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:20.556655 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d275207d-476a-4ce4-9ce2-da76a36e658d-tmp\") pod \"kube-auth-proxy-697457d895-hj6vs\" (UID: \"d275207d-476a-4ce4-9ce2-da76a36e658d\") " pod="openshift-ingress/kube-auth-proxy-697457d895-hj6vs" Jul 10 19:05:20.556856 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:20.556780 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d275207d-476a-4ce4-9ce2-da76a36e658d-tls-certs\") pod \"kube-auth-proxy-697457d895-hj6vs\" (UID: \"d275207d-476a-4ce4-9ce2-da76a36e658d\") " pod="openshift-ingress/kube-auth-proxy-697457d895-hj6vs" Jul 10 19:05:20.558948 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:20.558917 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d275207d-476a-4ce4-9ce2-da76a36e658d-tmp\") pod \"kube-auth-proxy-697457d895-hj6vs\" (UID: \"d275207d-476a-4ce4-9ce2-da76a36e658d\") " pod="openshift-ingress/kube-auth-proxy-697457d895-hj6vs" Jul 10 19:05:20.559145 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:20.559126 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d275207d-476a-4ce4-9ce2-da76a36e658d-tls-certs\") pod \"kube-auth-proxy-697457d895-hj6vs\" (UID: \"d275207d-476a-4ce4-9ce2-da76a36e658d\") " pod="openshift-ingress/kube-auth-proxy-697457d895-hj6vs" Jul 10 19:05:20.583275 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:20.583243 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-697457d895-hj6vs"] Jul 10 19:05:20.591472 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:20.591438 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lfkw\" (UniqueName: \"kubernetes.io/projected/d275207d-476a-4ce4-9ce2-da76a36e658d-kube-api-access-5lfkw\") pod \"kube-auth-proxy-697457d895-hj6vs\" (UID: \"d275207d-476a-4ce4-9ce2-da76a36e658d\") " pod="openshift-ingress/kube-auth-proxy-697457d895-hj6vs" Jul 10 19:05:20.636162 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:20.636061 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-9g4cr" Jul 10 19:05:20.665753 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:20.665723 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-697457d895-hj6vs" Jul 10 19:05:20.825166 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:20.825134 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-9g4cr"] Jul 10 19:05:20.828823 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:05:20.828794 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29957d41_1783_4280_bee2_75c4529ab426.slice/crio-de9ea39fd80170c095503c3018a2a1101f9ddccf95c948704d2fe993ddf2c97b WatchSource:0}: Error finding container de9ea39fd80170c095503c3018a2a1101f9ddccf95c948704d2fe993ddf2c97b: Status 404 returned error can't find the container with id de9ea39fd80170c095503c3018a2a1101f9ddccf95c948704d2fe993ddf2c97b Jul 10 19:05:20.875709 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:20.875667 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-697457d895-hj6vs"] Jul 10 19:05:20.878327 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:05:20.878303 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd275207d_476a_4ce4_9ce2_da76a36e658d.slice/crio-1b079901c5e29febcba46882ef5b95ee5f86ae0e11e5949ae43085cba72c5f24 WatchSource:0}: Error finding container 1b079901c5e29febcba46882ef5b95ee5f86ae0e11e5949ae43085cba72c5f24: Status 404 returned error can't find the container with id 1b079901c5e29febcba46882ef5b95ee5f86ae0e11e5949ae43085cba72c5f24 Jul 10 19:05:21.321904 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:21.321866 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-697457d895-hj6vs" event={"ID":"d275207d-476a-4ce4-9ce2-da76a36e658d","Type":"ContainerStarted","Data":"1b079901c5e29febcba46882ef5b95ee5f86ae0e11e5949ae43085cba72c5f24"} Jul 10 19:05:21.322964 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:21.322941 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-9g4cr" event={"ID":"29957d41-1783-4280-bee2-75c4529ab426","Type":"ContainerStarted","Data":"de9ea39fd80170c095503c3018a2a1101f9ddccf95c948704d2fe993ddf2c97b"} Jul 10 19:05:22.240257 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:22.239586 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-92hm5"] Jul 10 19:05:22.243749 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:22.243723 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-92hm5" Jul 10 19:05:22.246792 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:22.246765 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 10 19:05:22.270220 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:22.270170 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-92hm5"] Jul 10 19:05:22.381016 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:22.380837 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/249c9d5b-dc25-47cb-b492-cb0ca0805ea9-tls-certs\") pod \"model-serving-api-54d89bd5fc-92hm5\" (UID: \"249c9d5b-dc25-47cb-b492-cb0ca0805ea9\") " pod="opendatahub/model-serving-api-54d89bd5fc-92hm5" Jul 10 19:05:22.381016 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:22.380912 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgsth\" (UniqueName: \"kubernetes.io/projected/249c9d5b-dc25-47cb-b492-cb0ca0805ea9-kube-api-access-fgsth\") pod \"model-serving-api-54d89bd5fc-92hm5\" (UID: \"249c9d5b-dc25-47cb-b492-cb0ca0805ea9\") " pod="opendatahub/model-serving-api-54d89bd5fc-92hm5" Jul 10 19:05:22.482504 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:22.482049 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/249c9d5b-dc25-47cb-b492-cb0ca0805ea9-tls-certs\") pod \"model-serving-api-54d89bd5fc-92hm5\" (UID: \"249c9d5b-dc25-47cb-b492-cb0ca0805ea9\") " pod="opendatahub/model-serving-api-54d89bd5fc-92hm5" Jul 10 19:05:22.482504 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:22.482119 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgsth\" (UniqueName: \"kubernetes.io/projected/249c9d5b-dc25-47cb-b492-cb0ca0805ea9-kube-api-access-fgsth\") pod \"model-serving-api-54d89bd5fc-92hm5\" (UID: \"249c9d5b-dc25-47cb-b492-cb0ca0805ea9\") " pod="opendatahub/model-serving-api-54d89bd5fc-92hm5" Jul 10 19:05:22.485743 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:22.485712 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/249c9d5b-dc25-47cb-b492-cb0ca0805ea9-tls-certs\") pod \"model-serving-api-54d89bd5fc-92hm5\" (UID: \"249c9d5b-dc25-47cb-b492-cb0ca0805ea9\") " pod="opendatahub/model-serving-api-54d89bd5fc-92hm5" Jul 10 19:05:22.559863 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:22.559826 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgsth\" (UniqueName: \"kubernetes.io/projected/249c9d5b-dc25-47cb-b492-cb0ca0805ea9-kube-api-access-fgsth\") pod \"model-serving-api-54d89bd5fc-92hm5\" (UID: \"249c9d5b-dc25-47cb-b492-cb0ca0805ea9\") " pod="opendatahub/model-serving-api-54d89bd5fc-92hm5" Jul 10 19:05:22.857141 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:22.857046 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-92hm5" Jul 10 19:05:23.923050 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:23.922999 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-b8w6k"] Jul 10 19:05:23.928151 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:23.928119 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b8w6k" Jul 10 19:05:23.934197 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:23.934134 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 10 19:05:23.934197 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:23.934137 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 10 19:05:23.937047 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:23.937016 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-c8d7b\"" Jul 10 19:05:23.954387 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:23.954358 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 10 19:05:23.958915 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:23.958871 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-b8w6k"] Jul 10 19:05:23.960642 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:23.960615 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 10 19:05:24.096215 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:24.096178 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17fd2386-55ad-4035-89d7-806c1802be37-istio-token\") pod \"istiod-openshift-gateway-7574459656-b8w6k\" (UID: \"17fd2386-55ad-4035-89d7-806c1802be37\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b8w6k" Jul 10 19:05:24.096215 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:24.096218 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/17fd2386-55ad-4035-89d7-806c1802be37-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-b8w6k\" (UID: \"17fd2386-55ad-4035-89d7-806c1802be37\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b8w6k" Jul 10 19:05:24.096458 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:24.096321 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rpnj\" (UniqueName: \"kubernetes.io/projected/17fd2386-55ad-4035-89d7-806c1802be37-kube-api-access-6rpnj\") pod \"istiod-openshift-gateway-7574459656-b8w6k\" (UID: \"17fd2386-55ad-4035-89d7-806c1802be37\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b8w6k" Jul 10 19:05:24.096458 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:24.096366 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/17fd2386-55ad-4035-89d7-806c1802be37-cacerts\") pod \"istiod-openshift-gateway-7574459656-b8w6k\" (UID: \"17fd2386-55ad-4035-89d7-806c1802be37\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b8w6k" Jul 10 19:05:24.096458 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:24.096399 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/17fd2386-55ad-4035-89d7-806c1802be37-local-certs\") pod \"istiod-openshift-gateway-7574459656-b8w6k\" (UID: \"17fd2386-55ad-4035-89d7-806c1802be37\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b8w6k" Jul 10 19:05:24.096458 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:24.096452 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/17fd2386-55ad-4035-89d7-806c1802be37-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-b8w6k\" (UID: \"17fd2386-55ad-4035-89d7-806c1802be37\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b8w6k" Jul 10 19:05:24.096662 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:24.096493 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/17fd2386-55ad-4035-89d7-806c1802be37-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-b8w6k\" (UID: \"17fd2386-55ad-4035-89d7-806c1802be37\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b8w6k" Jul 10 19:05:24.197364 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:24.197273 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17fd2386-55ad-4035-89d7-806c1802be37-istio-token\") pod \"istiod-openshift-gateway-7574459656-b8w6k\" (UID: \"17fd2386-55ad-4035-89d7-806c1802be37\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b8w6k" Jul 10 19:05:24.197364 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:24.197336 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/17fd2386-55ad-4035-89d7-806c1802be37-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-b8w6k\" (UID: \"17fd2386-55ad-4035-89d7-806c1802be37\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b8w6k" Jul 10 19:05:24.197635 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:24.197398 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rpnj\" (UniqueName: \"kubernetes.io/projected/17fd2386-55ad-4035-89d7-806c1802be37-kube-api-access-6rpnj\") pod \"istiod-openshift-gateway-7574459656-b8w6k\" (UID: \"17fd2386-55ad-4035-89d7-806c1802be37\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b8w6k" Jul 10 19:05:24.197635 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:24.197431 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/17fd2386-55ad-4035-89d7-806c1802be37-cacerts\") pod \"istiod-openshift-gateway-7574459656-b8w6k\" (UID: \"17fd2386-55ad-4035-89d7-806c1802be37\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b8w6k" Jul 10 19:05:24.197635 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:24.197465 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/17fd2386-55ad-4035-89d7-806c1802be37-local-certs\") pod \"istiod-openshift-gateway-7574459656-b8w6k\" (UID: \"17fd2386-55ad-4035-89d7-806c1802be37\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b8w6k" Jul 10 19:05:24.197635 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:24.197503 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/17fd2386-55ad-4035-89d7-806c1802be37-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-b8w6k\" (UID: \"17fd2386-55ad-4035-89d7-806c1802be37\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b8w6k" Jul 10 19:05:24.197635 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:24.197544 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/17fd2386-55ad-4035-89d7-806c1802be37-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-b8w6k\" (UID: \"17fd2386-55ad-4035-89d7-806c1802be37\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b8w6k" Jul 10 19:05:24.198326 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:24.198277 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/17fd2386-55ad-4035-89d7-806c1802be37-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-b8w6k\" (UID: \"17fd2386-55ad-4035-89d7-806c1802be37\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b8w6k" Jul 10 19:05:24.200614 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:24.200588 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/17fd2386-55ad-4035-89d7-806c1802be37-cacerts\") pod \"istiod-openshift-gateway-7574459656-b8w6k\" (UID: \"17fd2386-55ad-4035-89d7-806c1802be37\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b8w6k" Jul 10 19:05:24.200749 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:24.200631 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/17fd2386-55ad-4035-89d7-806c1802be37-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-b8w6k\" (UID: \"17fd2386-55ad-4035-89d7-806c1802be37\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b8w6k" Jul 10 19:05:24.200749 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:24.200736 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/17fd2386-55ad-4035-89d7-806c1802be37-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-b8w6k\" (UID: \"17fd2386-55ad-4035-89d7-806c1802be37\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b8w6k" Jul 10 19:05:24.200877 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:24.200815 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/17fd2386-55ad-4035-89d7-806c1802be37-local-certs\") pod \"istiod-openshift-gateway-7574459656-b8w6k\" (UID: \"17fd2386-55ad-4035-89d7-806c1802be37\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b8w6k" Jul 10 19:05:24.215128 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:24.215096 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rpnj\" (UniqueName: \"kubernetes.io/projected/17fd2386-55ad-4035-89d7-806c1802be37-kube-api-access-6rpnj\") pod \"istiod-openshift-gateway-7574459656-b8w6k\" (UID: \"17fd2386-55ad-4035-89d7-806c1802be37\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b8w6k" Jul 10 19:05:24.220186 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:24.220154 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17fd2386-55ad-4035-89d7-806c1802be37-istio-token\") pod \"istiod-openshift-gateway-7574459656-b8w6k\" (UID: \"17fd2386-55ad-4035-89d7-806c1802be37\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b8w6k" Jul 10 19:05:24.240033 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:24.240004 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b8w6k" Jul 10 19:05:24.520327 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:24.520288 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-92hm5"] Jul 10 19:05:24.526252 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:24.526209 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-b8w6k"] Jul 10 19:05:24.530111 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:05:24.530083 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod249c9d5b_dc25_47cb_b492_cb0ca0805ea9.slice/crio-3dd577f941b62d131c64d0c231e5ee589fc8aeb138f96b3486b60c187172dea5 WatchSource:0}: Error finding container 3dd577f941b62d131c64d0c231e5ee589fc8aeb138f96b3486b60c187172dea5: Status 404 returned error can't find the container with id 3dd577f941b62d131c64d0c231e5ee589fc8aeb138f96b3486b60c187172dea5 Jul 10 19:05:24.530727 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:05:24.530701 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17fd2386_55ad_4035_89d7_806c1802be37.slice/crio-225012e6f2d74e30f0c2533f195e181f419ee4cbf1941db528c7e5b0b27c4dfa WatchSource:0}: Error finding container 225012e6f2d74e30f0c2533f195e181f419ee4cbf1941db528c7e5b0b27c4dfa: Status 404 returned error can't find the container with id 225012e6f2d74e30f0c2533f195e181f419ee4cbf1941db528c7e5b0b27c4dfa Jul 10 19:05:25.349424 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:25.349385 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-697457d895-hj6vs" event={"ID":"d275207d-476a-4ce4-9ce2-da76a36e658d","Type":"ContainerStarted","Data":"96b1cacf80afb87d91d1249edb2db06bfa696ea057341a7fa69b542e902128ff"} Jul 10 19:05:25.351201 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:25.351174 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-9g4cr" event={"ID":"29957d41-1783-4280-bee2-75c4529ab426","Type":"ContainerStarted","Data":"418dd847313df3f730e628d2b643dbb255f51b2a15138556f74b31155d7cfcf6"} Jul 10 19:05:25.351345 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:25.351315 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-9g4cr" Jul 10 19:05:25.352470 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:25.352450 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b8w6k" event={"ID":"17fd2386-55ad-4035-89d7-806c1802be37","Type":"ContainerStarted","Data":"225012e6f2d74e30f0c2533f195e181f419ee4cbf1941db528c7e5b0b27c4dfa"} Jul 10 19:05:25.353552 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:25.353533 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-92hm5" event={"ID":"249c9d5b-dc25-47cb-b492-cb0ca0805ea9","Type":"ContainerStarted","Data":"3dd577f941b62d131c64d0c231e5ee589fc8aeb138f96b3486b60c187172dea5"} Jul 10 19:05:25.462716 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:25.462640 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-697457d895-hj6vs" podStartSLOduration=2.03300559 podStartE2EDuration="5.462623215s" totalImagesPullingTime="3.429617625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:05:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:05:25.461734847 +0000 UTC m=+409.505877369" watchObservedRunningTime="2026-07-10 19:05:25.462623215 +0000 UTC m=+409.506765716" Jul 10 19:05:25.541935 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:25.541882 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-9g4cr" podStartSLOduration=3.064177183 podStartE2EDuration="6.541862634s" totalImagesPullingTime="3.477685451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:05:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:05:25.541345162 +0000 UTC m=+409.585487719" watchObservedRunningTime="2026-07-10 19:05:25.541862634 +0000 UTC m=+409.586005173" Jul 10 19:05:27.371235 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:27.371191 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-92hm5" event={"ID":"249c9d5b-dc25-47cb-b492-cb0ca0805ea9","Type":"ContainerStarted","Data":"e453ee9d824f100e585c64ce9fc7e7d2d85a9c502a719398515775072fd88bed"} Jul 10 19:05:27.371715 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:27.371275 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-92hm5" Jul 10 19:05:27.430333 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:27.430276 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-92hm5" podStartSLOduration=3.439750229 podStartE2EDuration="5.430255609s" totalImagesPullingTime="1.99050538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:05:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:05:27.429277855 +0000 UTC m=+411.473420383" watchObservedRunningTime="2026-07-10 19:05:27.430255609 +0000 UTC m=+411.474398111" Jul 10 19:05:27.819820 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:27.819760 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236236Ki","pods":"250"} Jul 10 19:05:27.819953 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:27.819851 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236236Ki","pods":"250"} Jul 10 19:05:28.379453 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:28.379404 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b8w6k" event={"ID":"17fd2386-55ad-4035-89d7-806c1802be37","Type":"ContainerStarted","Data":"25484c8b959a79936c1f8c5f53954480882d792f050aa4bdd1daa7efad68eea9"} Jul 10 19:05:28.380068 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:28.379676 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b8w6k" Jul 10 19:05:28.381550 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:28.381522 2581 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-b8w6k container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 10 19:05:28.381671 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:28.381593 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b8w6k" podUID="17fd2386-55ad-4035-89d7-806c1802be37" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 19:05:28.445996 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:28.445926 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b8w6k" podStartSLOduration=2.159395804 podStartE2EDuration="5.445904342s" totalImagesPullingTime="3.286508538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:05:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:05:28.439282901 +0000 UTC m=+412.483425412" watchObservedRunningTime="2026-07-10 19:05:28.445904342 +0000 UTC m=+412.490046857" Jul 10 19:05:29.383820 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:29.383788 2581 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-b8w6k container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 10 19:05:29.384199 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:29.383851 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b8w6k" podUID="17fd2386-55ad-4035-89d7-806c1802be37" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 19:05:32.384528 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:32.384498 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b8w6k" Jul 10 19:05:34.611456 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:34.611416 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-b4f45"] Jul 10 19:05:34.620117 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:34.620092 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-b4f45" Jul 10 19:05:34.622979 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:34.622955 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-gkhdl\"" Jul 10 19:05:34.623228 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:34.623209 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 10 19:05:34.637169 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:34.637135 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-kkqx7"] Jul 10 19:05:34.640185 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:34.640168 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kkqx7" Jul 10 19:05:34.643234 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:34.643213 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 10 19:05:34.643462 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:34.643439 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-zggjl\"" Jul 10 19:05:34.650396 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:34.650370 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-b4f45"] Jul 10 19:05:34.673607 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:34.673571 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-kkqx7"] Jul 10 19:05:34.689892 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:34.689836 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nj9k\" (UniqueName: \"kubernetes.io/projected/57d9470d-fca4-4f43-907d-3b40595570fb-kube-api-access-8nj9k\") pod \"llmisvc-controller-manager-777c6bd7d5-kkqx7\" (UID: \"57d9470d-fca4-4f43-907d-3b40595570fb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kkqx7" Jul 10 19:05:34.689892 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:34.689893 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/57d9470d-fca4-4f43-907d-3b40595570fb-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-kkqx7\" (UID: \"57d9470d-fca4-4f43-907d-3b40595570fb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kkqx7" Jul 10 19:05:34.690100 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:34.689982 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/112b1d22-d1b8-4de9-9baf-2821f4566806-cert\") pod \"kserve-controller-manager-d6f4b65b8-b4f45\" (UID: \"112b1d22-d1b8-4de9-9baf-2821f4566806\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-b4f45" Jul 10 19:05:34.690157 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:34.690093 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74p9t\" (UniqueName: \"kubernetes.io/projected/112b1d22-d1b8-4de9-9baf-2821f4566806-kube-api-access-74p9t\") pod \"kserve-controller-manager-d6f4b65b8-b4f45\" (UID: \"112b1d22-d1b8-4de9-9baf-2821f4566806\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-b4f45" Jul 10 19:05:34.791431 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:34.791378 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nj9k\" (UniqueName: \"kubernetes.io/projected/57d9470d-fca4-4f43-907d-3b40595570fb-kube-api-access-8nj9k\") pod \"llmisvc-controller-manager-777c6bd7d5-kkqx7\" (UID: \"57d9470d-fca4-4f43-907d-3b40595570fb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kkqx7" Jul 10 19:05:34.791431 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:34.791434 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/57d9470d-fca4-4f43-907d-3b40595570fb-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-kkqx7\" (UID: \"57d9470d-fca4-4f43-907d-3b40595570fb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kkqx7" Jul 10 19:05:34.791659 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:34.791462 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/112b1d22-d1b8-4de9-9baf-2821f4566806-cert\") pod \"kserve-controller-manager-d6f4b65b8-b4f45\" (UID: \"112b1d22-d1b8-4de9-9baf-2821f4566806\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-b4f45" Jul 10 19:05:34.791659 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:34.791507 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74p9t\" (UniqueName: \"kubernetes.io/projected/112b1d22-d1b8-4de9-9baf-2821f4566806-kube-api-access-74p9t\") pod \"kserve-controller-manager-d6f4b65b8-b4f45\" (UID: \"112b1d22-d1b8-4de9-9baf-2821f4566806\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-b4f45" Jul 10 19:05:34.791659 ip-10-0-135-67 kubenswrapper[2581]: E0710 19:05:34.791608 2581 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 10 19:05:34.791803 ip-10-0-135-67 kubenswrapper[2581]: E0710 19:05:34.791609 2581 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 10 19:05:34.791803 ip-10-0-135-67 kubenswrapper[2581]: E0710 19:05:34.791680 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57d9470d-fca4-4f43-907d-3b40595570fb-cert podName:57d9470d-fca4-4f43-907d-3b40595570fb nodeName:}" failed. No retries permitted until 2026-07-10 19:05:35.291659908 +0000 UTC m=+419.335802402 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/57d9470d-fca4-4f43-907d-3b40595570fb-cert") pod "llmisvc-controller-manager-777c6bd7d5-kkqx7" (UID: "57d9470d-fca4-4f43-907d-3b40595570fb") : secret "llmisvc-webhook-server-cert" not found Jul 10 19:05:34.791803 ip-10-0-135-67 kubenswrapper[2581]: E0710 19:05:34.791799 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/112b1d22-d1b8-4de9-9baf-2821f4566806-cert podName:112b1d22-d1b8-4de9-9baf-2821f4566806 nodeName:}" failed. No retries permitted until 2026-07-10 19:05:35.291783764 +0000 UTC m=+419.335926248 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/112b1d22-d1b8-4de9-9baf-2821f4566806-cert") pod "kserve-controller-manager-d6f4b65b8-b4f45" (UID: "112b1d22-d1b8-4de9-9baf-2821f4566806") : secret "kserve-webhook-server-cert" not found Jul 10 19:05:34.806866 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:34.806836 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nj9k\" (UniqueName: \"kubernetes.io/projected/57d9470d-fca4-4f43-907d-3b40595570fb-kube-api-access-8nj9k\") pod \"llmisvc-controller-manager-777c6bd7d5-kkqx7\" (UID: \"57d9470d-fca4-4f43-907d-3b40595570fb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kkqx7" Jul 10 19:05:34.808261 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:34.808238 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74p9t\" (UniqueName: \"kubernetes.io/projected/112b1d22-d1b8-4de9-9baf-2821f4566806-kube-api-access-74p9t\") pod \"kserve-controller-manager-d6f4b65b8-b4f45\" (UID: \"112b1d22-d1b8-4de9-9baf-2821f4566806\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-b4f45" Jul 10 19:05:35.296738 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:35.296662 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/57d9470d-fca4-4f43-907d-3b40595570fb-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-kkqx7\" (UID: \"57d9470d-fca4-4f43-907d-3b40595570fb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kkqx7" Jul 10 19:05:35.296936 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:35.296760 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/112b1d22-d1b8-4de9-9baf-2821f4566806-cert\") pod \"kserve-controller-manager-d6f4b65b8-b4f45\" (UID: \"112b1d22-d1b8-4de9-9baf-2821f4566806\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-b4f45" Jul 10 19:05:35.299212 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:35.299179 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/112b1d22-d1b8-4de9-9baf-2821f4566806-cert\") pod \"kserve-controller-manager-d6f4b65b8-b4f45\" (UID: \"112b1d22-d1b8-4de9-9baf-2821f4566806\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-b4f45" Jul 10 19:05:35.299327 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:35.299247 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/57d9470d-fca4-4f43-907d-3b40595570fb-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-kkqx7\" (UID: \"57d9470d-fca4-4f43-907d-3b40595570fb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kkqx7" Jul 10 19:05:35.530323 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:35.530288 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-b4f45" Jul 10 19:05:35.550522 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:35.550452 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kkqx7" Jul 10 19:05:35.707702 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:35.707638 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-kkqx7"] Jul 10 19:05:35.708052 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:05:35.707993 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod112b1d22_d1b8_4de9_9baf_2821f4566806.slice/crio-9197084a0655d3967fcdffc7f9b17a4e2ef5c923574608cce4f3b407e636c2b4 WatchSource:0}: Error finding container 9197084a0655d3967fcdffc7f9b17a4e2ef5c923574608cce4f3b407e636c2b4: Status 404 returned error can't find the container with id 9197084a0655d3967fcdffc7f9b17a4e2ef5c923574608cce4f3b407e636c2b4 Jul 10 19:05:35.709614 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:35.709586 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-b4f45"] Jul 10 19:05:35.709916 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:05:35.709889 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod57d9470d_fca4_4f43_907d_3b40595570fb.slice/crio-600a5f66ddaf1618729ec5ba1bce5358fdb56bc0aac2b2ad4c2d2f66301e1f96 WatchSource:0}: Error finding container 600a5f66ddaf1618729ec5ba1bce5358fdb56bc0aac2b2ad4c2d2f66301e1f96: Status 404 returned error can't find the container with id 600a5f66ddaf1618729ec5ba1bce5358fdb56bc0aac2b2ad4c2d2f66301e1f96 Jul 10 19:05:36.363161 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:36.363128 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-9g4cr" Jul 10 19:05:36.423619 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:36.423551 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-b4f45" event={"ID":"112b1d22-d1b8-4de9-9baf-2821f4566806","Type":"ContainerStarted","Data":"9197084a0655d3967fcdffc7f9b17a4e2ef5c923574608cce4f3b407e636c2b4"} Jul 10 19:05:36.425430 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:36.425405 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kkqx7" event={"ID":"57d9470d-fca4-4f43-907d-3b40595570fb","Type":"ContainerStarted","Data":"600a5f66ddaf1618729ec5ba1bce5358fdb56bc0aac2b2ad4c2d2f66301e1f96"} Jul 10 19:05:37.298198 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:37.298158 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-b8w6k"] Jul 10 19:05:37.298575 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:37.298428 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b8w6k" podUID="17fd2386-55ad-4035-89d7-806c1802be37" containerName="discovery" containerID="cri-o://25484c8b959a79936c1f8c5f53954480882d792f050aa4bdd1daa7efad68eea9" gracePeriod=30 Jul 10 19:05:37.433612 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:37.433571 2581 generic.go:356] "Generic (PLEG): container finished" podID="17fd2386-55ad-4035-89d7-806c1802be37" containerID="25484c8b959a79936c1f8c5f53954480882d792f050aa4bdd1daa7efad68eea9" exitCode=0 Jul 10 19:05:37.433802 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:37.433671 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b8w6k" event={"ID":"17fd2386-55ad-4035-89d7-806c1802be37","Type":"ContainerDied","Data":"25484c8b959a79936c1f8c5f53954480882d792f050aa4bdd1daa7efad68eea9"} Jul 10 19:05:37.569839 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:37.569813 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b8w6k" Jul 10 19:05:37.619156 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:37.619120 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/17fd2386-55ad-4035-89d7-806c1802be37-local-certs\") pod \"17fd2386-55ad-4035-89d7-806c1802be37\" (UID: \"17fd2386-55ad-4035-89d7-806c1802be37\") " Jul 10 19:05:37.619340 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:37.619172 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rpnj\" (UniqueName: \"kubernetes.io/projected/17fd2386-55ad-4035-89d7-806c1802be37-kube-api-access-6rpnj\") pod \"17fd2386-55ad-4035-89d7-806c1802be37\" (UID: \"17fd2386-55ad-4035-89d7-806c1802be37\") " Jul 10 19:05:37.619340 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:37.619240 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/17fd2386-55ad-4035-89d7-806c1802be37-istio-csr-dns-cert\") pod \"17fd2386-55ad-4035-89d7-806c1802be37\" (UID: \"17fd2386-55ad-4035-89d7-806c1802be37\") " Jul 10 19:05:37.619340 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:37.619266 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17fd2386-55ad-4035-89d7-806c1802be37-istio-token\") pod \"17fd2386-55ad-4035-89d7-806c1802be37\" (UID: \"17fd2386-55ad-4035-89d7-806c1802be37\") " Jul 10 19:05:37.619340 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:37.619302 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/17fd2386-55ad-4035-89d7-806c1802be37-istio-kubeconfig\") pod \"17fd2386-55ad-4035-89d7-806c1802be37\" (UID: \"17fd2386-55ad-4035-89d7-806c1802be37\") " Jul 10 19:05:37.619340 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:37.619324 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/17fd2386-55ad-4035-89d7-806c1802be37-cacerts\") pod \"17fd2386-55ad-4035-89d7-806c1802be37\" (UID: \"17fd2386-55ad-4035-89d7-806c1802be37\") " Jul 10 19:05:37.619613 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:37.619351 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/17fd2386-55ad-4035-89d7-806c1802be37-istio-csr-ca-configmap\") pod \"17fd2386-55ad-4035-89d7-806c1802be37\" (UID: \"17fd2386-55ad-4035-89d7-806c1802be37\") " Jul 10 19:05:37.620315 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:37.620281 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17fd2386-55ad-4035-89d7-806c1802be37-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "17fd2386-55ad-4035-89d7-806c1802be37" (UID: "17fd2386-55ad-4035-89d7-806c1802be37"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:05:37.623246 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:37.623201 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17fd2386-55ad-4035-89d7-806c1802be37-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "17fd2386-55ad-4035-89d7-806c1802be37" (UID: "17fd2386-55ad-4035-89d7-806c1802be37"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:05:37.625447 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:37.625407 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17fd2386-55ad-4035-89d7-806c1802be37-local-certs" (OuterVolumeSpecName: "local-certs") pod "17fd2386-55ad-4035-89d7-806c1802be37" (UID: "17fd2386-55ad-4035-89d7-806c1802be37"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:05:37.625553 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:37.625444 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17fd2386-55ad-4035-89d7-806c1802be37-istio-token" (OuterVolumeSpecName: "istio-token") pod "17fd2386-55ad-4035-89d7-806c1802be37" (UID: "17fd2386-55ad-4035-89d7-806c1802be37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:05:37.625614 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:37.625596 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17fd2386-55ad-4035-89d7-806c1802be37-kube-api-access-6rpnj" (OuterVolumeSpecName: "kube-api-access-6rpnj") pod "17fd2386-55ad-4035-89d7-806c1802be37" (UID: "17fd2386-55ad-4035-89d7-806c1802be37"). InnerVolumeSpecName "kube-api-access-6rpnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:05:37.626772 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:37.626741 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17fd2386-55ad-4035-89d7-806c1802be37-cacerts" (OuterVolumeSpecName: "cacerts") pod "17fd2386-55ad-4035-89d7-806c1802be37" (UID: "17fd2386-55ad-4035-89d7-806c1802be37"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:05:37.627201 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:37.627166 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17fd2386-55ad-4035-89d7-806c1802be37-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "17fd2386-55ad-4035-89d7-806c1802be37" (UID: "17fd2386-55ad-4035-89d7-806c1802be37"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:05:37.720440 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:37.720397 2581 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/17fd2386-55ad-4035-89d7-806c1802be37-local-certs\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:05:37.720440 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:37.720437 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rpnj\" (UniqueName: \"kubernetes.io/projected/17fd2386-55ad-4035-89d7-806c1802be37-kube-api-access-6rpnj\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:05:37.720440 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:37.720447 2581 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/17fd2386-55ad-4035-89d7-806c1802be37-istio-csr-dns-cert\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:05:37.720440 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:37.720456 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17fd2386-55ad-4035-89d7-806c1802be37-istio-token\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:05:37.720440 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:37.720467 2581 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/17fd2386-55ad-4035-89d7-806c1802be37-istio-kubeconfig\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:05:37.720440 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:37.720480 2581 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/17fd2386-55ad-4035-89d7-806c1802be37-cacerts\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:05:37.720440 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:37.720492 2581 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/17fd2386-55ad-4035-89d7-806c1802be37-istio-csr-ca-configmap\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:05:38.384037 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:38.384003 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-92hm5" Jul 10 19:05:38.441108 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:38.441070 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b8w6k" event={"ID":"17fd2386-55ad-4035-89d7-806c1802be37","Type":"ContainerDied","Data":"225012e6f2d74e30f0c2533f195e181f419ee4cbf1941db528c7e5b0b27c4dfa"} Jul 10 19:05:38.441277 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:38.441125 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b8w6k" Jul 10 19:05:38.441277 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:38.441162 2581 scope.go:117] "RemoveContainer" containerID="25484c8b959a79936c1f8c5f53954480882d792f050aa4bdd1daa7efad68eea9" Jul 10 19:05:38.472464 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:38.472436 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-b8w6k"] Jul 10 19:05:38.476560 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:38.476534 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-b8w6k"] Jul 10 19:05:38.544104 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:38.544070 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17fd2386-55ad-4035-89d7-806c1802be37" path="/var/lib/kubelet/pods/17fd2386-55ad-4035-89d7-806c1802be37/volumes" Jul 10 19:05:39.449107 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:39.449061 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kkqx7" event={"ID":"57d9470d-fca4-4f43-907d-3b40595570fb","Type":"ContainerStarted","Data":"d556241756f5773003cf732fb085127a9fb97150e8f7b206c735c29696ef5e6b"} Jul 10 19:05:39.449550 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:39.449124 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kkqx7" Jul 10 19:05:39.450604 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:39.450579 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-b4f45" event={"ID":"112b1d22-d1b8-4de9-9baf-2821f4566806","Type":"ContainerStarted","Data":"75caf5a2a6648a905cd09cf4b6242dab7ff731d5b74f85d93741825d079e58df"} Jul 10 19:05:39.450740 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:39.450642 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-b4f45" Jul 10 19:05:39.468716 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:39.468648 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kkqx7" podStartSLOduration=2.247226921 podStartE2EDuration="5.468632614s" totalImagesPullingTime="3.221405693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:05:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:05:39.467572105 +0000 UTC m=+423.511714624" watchObservedRunningTime="2026-07-10 19:05:39.468632614 +0000 UTC m=+423.512775115" Jul 10 19:05:39.489855 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:05:39.489805 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-b4f45" podStartSLOduration=2.270180449 podStartE2EDuration="5.489789094s" totalImagesPullingTime="3.219608645s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:05:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:05:39.488277674 +0000 UTC m=+423.532420174" watchObservedRunningTime="2026-07-10 19:05:39.489789094 +0000 UTC m=+423.533931593" Jul 10 19:06:10.458596 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:10.458507 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kkqx7" Jul 10 19:06:10.461389 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:10.461363 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-b4f45" Jul 10 19:06:20.152895 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:20.152844 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kjbvd"] Jul 10 19:06:20.153267 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:20.153187 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17fd2386-55ad-4035-89d7-806c1802be37" containerName="discovery" Jul 10 19:06:20.153267 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:20.153198 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="17fd2386-55ad-4035-89d7-806c1802be37" containerName="discovery" Jul 10 19:06:20.153267 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:20.153246 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="17fd2386-55ad-4035-89d7-806c1802be37" containerName="discovery" Jul 10 19:06:20.159887 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:20.159832 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-kjbvd" Jul 10 19:06:20.162926 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:20.162901 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 10 19:06:20.163066 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:20.162903 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-cgsz2\"" Jul 10 19:06:20.163543 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:20.163525 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 10 19:06:20.168401 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:20.168374 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kjbvd"] Jul 10 19:06:20.180599 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:20.180566 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2m8k\" (UniqueName: \"kubernetes.io/projected/e2d13a15-39c7-410b-8839-81f1a1bc29af-kube-api-access-h2m8k\") pod \"kuadrant-operator-catalog-kjbvd\" (UID: \"e2d13a15-39c7-410b-8839-81f1a1bc29af\") " pod="kuadrant-system/kuadrant-operator-catalog-kjbvd" Jul 10 19:06:20.281132 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:20.281093 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2m8k\" (UniqueName: \"kubernetes.io/projected/e2d13a15-39c7-410b-8839-81f1a1bc29af-kube-api-access-h2m8k\") pod \"kuadrant-operator-catalog-kjbvd\" (UID: \"e2d13a15-39c7-410b-8839-81f1a1bc29af\") " pod="kuadrant-system/kuadrant-operator-catalog-kjbvd" Jul 10 19:06:20.304076 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:20.304045 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2m8k\" (UniqueName: \"kubernetes.io/projected/e2d13a15-39c7-410b-8839-81f1a1bc29af-kube-api-access-h2m8k\") pod \"kuadrant-operator-catalog-kjbvd\" (UID: \"e2d13a15-39c7-410b-8839-81f1a1bc29af\") " pod="kuadrant-system/kuadrant-operator-catalog-kjbvd" Jul 10 19:06:20.471789 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:20.471704 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-kjbvd" Jul 10 19:06:20.482534 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:20.482509 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kjbvd"] Jul 10 19:06:20.603505 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:20.603481 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kjbvd"] Jul 10 19:06:20.605577 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:06:20.605550 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2d13a15_39c7_410b_8839_81f1a1bc29af.slice/crio-3d4ef275c26faac4ab24d0c87960b97aac6d127ffdf92a31a3058fe0d33b2a08 WatchSource:0}: Error finding container 3d4ef275c26faac4ab24d0c87960b97aac6d127ffdf92a31a3058fe0d33b2a08: Status 404 returned error can't find the container with id 3d4ef275c26faac4ab24d0c87960b97aac6d127ffdf92a31a3058fe0d33b2a08 Jul 10 19:06:20.636148 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:20.636119 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-kjbvd" event={"ID":"e2d13a15-39c7-410b-8839-81f1a1bc29af","Type":"ContainerStarted","Data":"3d4ef275c26faac4ab24d0c87960b97aac6d127ffdf92a31a3058fe0d33b2a08"} Jul 10 19:06:20.694891 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:20.694854 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wwz8v"] Jul 10 19:06:20.698173 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:20.698151 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-wwz8v" Jul 10 19:06:20.707935 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:20.707912 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wwz8v"] Jul 10 19:06:20.783817 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:20.783773 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8znd7\" (UniqueName: \"kubernetes.io/projected/670aa800-3f0d-4c74-830f-ac82faad07c9-kube-api-access-8znd7\") pod \"kuadrant-operator-catalog-wwz8v\" (UID: \"670aa800-3f0d-4c74-830f-ac82faad07c9\") " pod="kuadrant-system/kuadrant-operator-catalog-wwz8v" Jul 10 19:06:20.884420 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:20.884385 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8znd7\" (UniqueName: \"kubernetes.io/projected/670aa800-3f0d-4c74-830f-ac82faad07c9-kube-api-access-8znd7\") pod \"kuadrant-operator-catalog-wwz8v\" (UID: \"670aa800-3f0d-4c74-830f-ac82faad07c9\") " pod="kuadrant-system/kuadrant-operator-catalog-wwz8v" Jul 10 19:06:20.894048 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:20.894023 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8znd7\" (UniqueName: \"kubernetes.io/projected/670aa800-3f0d-4c74-830f-ac82faad07c9-kube-api-access-8znd7\") pod \"kuadrant-operator-catalog-wwz8v\" (UID: \"670aa800-3f0d-4c74-830f-ac82faad07c9\") " pod="kuadrant-system/kuadrant-operator-catalog-wwz8v" Jul 10 19:06:21.007860 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:21.007826 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-wwz8v" Jul 10 19:06:21.203629 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:21.203593 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wwz8v"] Jul 10 19:06:21.246833 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:06:21.246800 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod670aa800_3f0d_4c74_830f_ac82faad07c9.slice/crio-527b762cb7dd12923523a7e4ada079b8f2ed82a3ff9ebfb792e0afe4555e7696 WatchSource:0}: Error finding container 527b762cb7dd12923523a7e4ada079b8f2ed82a3ff9ebfb792e0afe4555e7696: Status 404 returned error can't find the container with id 527b762cb7dd12923523a7e4ada079b8f2ed82a3ff9ebfb792e0afe4555e7696 Jul 10 19:06:21.643450 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:21.643411 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-wwz8v" event={"ID":"670aa800-3f0d-4c74-830f-ac82faad07c9","Type":"ContainerStarted","Data":"527b762cb7dd12923523a7e4ada079b8f2ed82a3ff9ebfb792e0afe4555e7696"} Jul 10 19:06:22.650252 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:22.650213 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-wwz8v" event={"ID":"670aa800-3f0d-4c74-830f-ac82faad07c9","Type":"ContainerStarted","Data":"c138036c4a8b48e4a4cc20291faf687f8c03070cf1ef9d3f0bc60421c798ff56"} Jul 10 19:06:22.651735 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:22.651706 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-kjbvd" event={"ID":"e2d13a15-39c7-410b-8839-81f1a1bc29af","Type":"ContainerStarted","Data":"b5dba09d73907daf03974e29eb305fa5ac63e6a591ae2d51d39a61ef9344915e"} Jul 10 19:06:22.651852 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:22.651737 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-kjbvd" podUID="e2d13a15-39c7-410b-8839-81f1a1bc29af" containerName="registry-server" containerID="cri-o://b5dba09d73907daf03974e29eb305fa5ac63e6a591ae2d51d39a61ef9344915e" gracePeriod=2 Jul 10 19:06:22.674022 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:22.673977 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-wwz8v" podStartSLOduration=1.470991248 podStartE2EDuration="2.673960358s" totalImagesPullingTime="1.20296911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:06:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:06:22.673317044 +0000 UTC m=+466.717459545" watchObservedRunningTime="2026-07-10 19:06:22.673960358 +0000 UTC m=+466.718102837" Jul 10 19:06:22.694945 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:22.694868 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-kjbvd" podStartSLOduration=0.851376602 podStartE2EDuration="2.694852104s" totalImagesPullingTime="1.843475502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:06:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:06:22.691873195 +0000 UTC m=+466.736015698" watchObservedRunningTime="2026-07-10 19:06:22.694852104 +0000 UTC m=+466.738994604" Jul 10 19:06:22.898208 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:22.898183 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-kjbvd" Jul 10 19:06:23.000935 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:23.000906 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2m8k\" (UniqueName: \"kubernetes.io/projected/e2d13a15-39c7-410b-8839-81f1a1bc29af-kube-api-access-h2m8k\") pod \"e2d13a15-39c7-410b-8839-81f1a1bc29af\" (UID: \"e2d13a15-39c7-410b-8839-81f1a1bc29af\") " Jul 10 19:06:23.003106 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:23.003081 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2d13a15-39c7-410b-8839-81f1a1bc29af-kube-api-access-h2m8k" (OuterVolumeSpecName: "kube-api-access-h2m8k") pod "e2d13a15-39c7-410b-8839-81f1a1bc29af" (UID: "e2d13a15-39c7-410b-8839-81f1a1bc29af"). InnerVolumeSpecName "kube-api-access-h2m8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:06:23.101769 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:23.101734 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2m8k\" (UniqueName: \"kubernetes.io/projected/e2d13a15-39c7-410b-8839-81f1a1bc29af-kube-api-access-h2m8k\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:06:23.657428 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:23.657393 2581 generic.go:356] "Generic (PLEG): container finished" podID="e2d13a15-39c7-410b-8839-81f1a1bc29af" containerID="b5dba09d73907daf03974e29eb305fa5ac63e6a591ae2d51d39a61ef9344915e" exitCode=0 Jul 10 19:06:23.657929 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:23.657449 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-kjbvd" Jul 10 19:06:23.657929 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:23.657488 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-kjbvd" event={"ID":"e2d13a15-39c7-410b-8839-81f1a1bc29af","Type":"ContainerDied","Data":"b5dba09d73907daf03974e29eb305fa5ac63e6a591ae2d51d39a61ef9344915e"} Jul 10 19:06:23.657929 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:23.657542 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-kjbvd" event={"ID":"e2d13a15-39c7-410b-8839-81f1a1bc29af","Type":"ContainerDied","Data":"3d4ef275c26faac4ab24d0c87960b97aac6d127ffdf92a31a3058fe0d33b2a08"} Jul 10 19:06:23.657929 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:23.657564 2581 scope.go:117] "RemoveContainer" containerID="b5dba09d73907daf03974e29eb305fa5ac63e6a591ae2d51d39a61ef9344915e" Jul 10 19:06:23.667046 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:23.667025 2581 scope.go:117] "RemoveContainer" containerID="b5dba09d73907daf03974e29eb305fa5ac63e6a591ae2d51d39a61ef9344915e" Jul 10 19:06:23.667319 ip-10-0-135-67 kubenswrapper[2581]: E0710 19:06:23.667299 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5dba09d73907daf03974e29eb305fa5ac63e6a591ae2d51d39a61ef9344915e\": container with ID starting with b5dba09d73907daf03974e29eb305fa5ac63e6a591ae2d51d39a61ef9344915e not found: ID does not exist" containerID="b5dba09d73907daf03974e29eb305fa5ac63e6a591ae2d51d39a61ef9344915e" Jul 10 19:06:23.667372 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:23.667330 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5dba09d73907daf03974e29eb305fa5ac63e6a591ae2d51d39a61ef9344915e"} err="failed to get container status \"b5dba09d73907daf03974e29eb305fa5ac63e6a591ae2d51d39a61ef9344915e\": rpc error: code = NotFound desc = could not find container \"b5dba09d73907daf03974e29eb305fa5ac63e6a591ae2d51d39a61ef9344915e\": container with ID starting with b5dba09d73907daf03974e29eb305fa5ac63e6a591ae2d51d39a61ef9344915e not found: ID does not exist" Jul 10 19:06:23.680085 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:23.680059 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kjbvd"] Jul 10 19:06:23.686224 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:23.686199 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kjbvd"] Jul 10 19:06:24.542096 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:24.542060 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2d13a15-39c7-410b-8839-81f1a1bc29af" path="/var/lib/kubelet/pods/e2d13a15-39c7-410b-8839-81f1a1bc29af/volumes" Jul 10 19:06:31.008385 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:31.008344 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-wwz8v" Jul 10 19:06:31.008813 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:31.008426 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-wwz8v" Jul 10 19:06:31.032064 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:31.032037 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-wwz8v" Jul 10 19:06:31.717424 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:31.717398 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-wwz8v" Jul 10 19:06:35.073953 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:35.073915 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gx4nh"] Jul 10 19:06:35.074423 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:35.074403 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2d13a15-39c7-410b-8839-81f1a1bc29af" containerName="registry-server" Jul 10 19:06:35.074491 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:35.074426 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2d13a15-39c7-410b-8839-81f1a1bc29af" containerName="registry-server" Jul 10 19:06:35.074569 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:35.074529 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2d13a15-39c7-410b-8839-81f1a1bc29af" containerName="registry-server" Jul 10 19:06:35.081610 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:35.081579 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gx4nh" Jul 10 19:06:35.085269 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:35.085107 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-4d9qg\"" Jul 10 19:06:35.086735 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:35.086707 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gx4nh"] Jul 10 19:06:35.096659 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:35.096618 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdz4l\" (UniqueName: \"kubernetes.io/projected/8b0dd963-9313-419e-a83c-72908d524086-kube-api-access-jdz4l\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gx4nh\" (UID: \"8b0dd963-9313-419e-a83c-72908d524086\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gx4nh" Jul 10 19:06:35.096798 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:35.096673 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8b0dd963-9313-419e-a83c-72908d524086-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gx4nh\" (UID: \"8b0dd963-9313-419e-a83c-72908d524086\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gx4nh" Jul 10 19:06:35.096798 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:35.096707 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8b0dd963-9313-419e-a83c-72908d524086-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gx4nh\" (UID: \"8b0dd963-9313-419e-a83c-72908d524086\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gx4nh" Jul 10 19:06:35.197241 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:35.197200 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdz4l\" (UniqueName: \"kubernetes.io/projected/8b0dd963-9313-419e-a83c-72908d524086-kube-api-access-jdz4l\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gx4nh\" (UID: \"8b0dd963-9313-419e-a83c-72908d524086\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gx4nh" Jul 10 19:06:35.197436 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:35.197254 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8b0dd963-9313-419e-a83c-72908d524086-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gx4nh\" (UID: \"8b0dd963-9313-419e-a83c-72908d524086\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gx4nh" Jul 10 19:06:35.197436 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:35.197287 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8b0dd963-9313-419e-a83c-72908d524086-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gx4nh\" (UID: \"8b0dd963-9313-419e-a83c-72908d524086\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gx4nh" Jul 10 19:06:35.197739 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:35.197720 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8b0dd963-9313-419e-a83c-72908d524086-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gx4nh\" (UID: \"8b0dd963-9313-419e-a83c-72908d524086\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gx4nh" Jul 10 19:06:35.197826 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:35.197775 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8b0dd963-9313-419e-a83c-72908d524086-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gx4nh\" (UID: \"8b0dd963-9313-419e-a83c-72908d524086\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gx4nh" Jul 10 19:06:35.207061 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:35.207036 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdz4l\" (UniqueName: \"kubernetes.io/projected/8b0dd963-9313-419e-a83c-72908d524086-kube-api-access-jdz4l\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gx4nh\" (UID: \"8b0dd963-9313-419e-a83c-72908d524086\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gx4nh" Jul 10 19:06:35.392247 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:35.392163 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gx4nh" Jul 10 19:06:35.480660 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:35.480627 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef156snh"] Jul 10 19:06:35.486031 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:35.485992 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef156snh" Jul 10 19:06:35.499130 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:35.498948 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkhb4\" (UniqueName: \"kubernetes.io/projected/ec050446-3850-4b65-bcf8-2e7bf2688638-kube-api-access-dkhb4\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef156snh\" (UID: \"ec050446-3850-4b65-bcf8-2e7bf2688638\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef156snh" Jul 10 19:06:35.499130 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:35.499011 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec050446-3850-4b65-bcf8-2e7bf2688638-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef156snh\" (UID: \"ec050446-3850-4b65-bcf8-2e7bf2688638\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef156snh" Jul 10 19:06:35.499130 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:35.499072 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec050446-3850-4b65-bcf8-2e7bf2688638-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef156snh\" (UID: \"ec050446-3850-4b65-bcf8-2e7bf2688638\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef156snh" Jul 10 19:06:35.501641 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:35.501605 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef156snh"] Jul 10 19:06:35.563742 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:35.563716 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gx4nh"] Jul 10 19:06:35.566172 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:06:35.566141 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b0dd963_9313_419e_a83c_72908d524086.slice/crio-202b123f66bbbee2ea5dcf86fc5ce4b061a86ccb8178fa5d1aac78efbe418cef WatchSource:0}: Error finding container 202b123f66bbbee2ea5dcf86fc5ce4b061a86ccb8178fa5d1aac78efbe418cef: Status 404 returned error can't find the container with id 202b123f66bbbee2ea5dcf86fc5ce4b061a86ccb8178fa5d1aac78efbe418cef Jul 10 19:06:35.599592 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:35.599569 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkhb4\" (UniqueName: \"kubernetes.io/projected/ec050446-3850-4b65-bcf8-2e7bf2688638-kube-api-access-dkhb4\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef156snh\" (UID: \"ec050446-3850-4b65-bcf8-2e7bf2688638\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef156snh" Jul 10 19:06:35.599719 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:35.599606 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec050446-3850-4b65-bcf8-2e7bf2688638-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef156snh\" (UID: \"ec050446-3850-4b65-bcf8-2e7bf2688638\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef156snh" Jul 10 19:06:35.599719 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:35.599641 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec050446-3850-4b65-bcf8-2e7bf2688638-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef156snh\" (UID: \"ec050446-3850-4b65-bcf8-2e7bf2688638\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef156snh" Jul 10 19:06:35.600065 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:35.600047 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec050446-3850-4b65-bcf8-2e7bf2688638-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef156snh\" (UID: \"ec050446-3850-4b65-bcf8-2e7bf2688638\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef156snh" Jul 10 19:06:35.600122 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:35.600064 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec050446-3850-4b65-bcf8-2e7bf2688638-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef156snh\" (UID: \"ec050446-3850-4b65-bcf8-2e7bf2688638\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef156snh" Jul 10 19:06:35.624714 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:35.624659 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkhb4\" (UniqueName: \"kubernetes.io/projected/ec050446-3850-4b65-bcf8-2e7bf2688638-kube-api-access-dkhb4\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef156snh\" (UID: \"ec050446-3850-4b65-bcf8-2e7bf2688638\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef156snh" Jul 10 19:06:35.714599 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:35.714568 2581 generic.go:356] "Generic (PLEG): container finished" podID="8b0dd963-9313-419e-a83c-72908d524086" containerID="f2a5ea2e7e694b6dce3ce28e4433bf44445fc861e8cda8646a70f3cea13a51ad" exitCode=0 Jul 10 19:06:35.714785 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:35.714654 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gx4nh" event={"ID":"8b0dd963-9313-419e-a83c-72908d524086","Type":"ContainerDied","Data":"f2a5ea2e7e694b6dce3ce28e4433bf44445fc861e8cda8646a70f3cea13a51ad"} Jul 10 19:06:35.714785 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:35.714731 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gx4nh" event={"ID":"8b0dd963-9313-419e-a83c-72908d524086","Type":"ContainerStarted","Data":"202b123f66bbbee2ea5dcf86fc5ce4b061a86ccb8178fa5d1aac78efbe418cef"} Jul 10 19:06:35.799041 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:35.799007 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef156snh" Jul 10 19:06:35.938352 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:35.938322 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef156snh"] Jul 10 19:06:35.940213 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:06:35.940186 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec050446_3850_4b65_bcf8_2e7bf2688638.slice/crio-0e30ba34428ca4b575a16dd74ec9b7b1586b54393ffd8a48d4347031b7edda5d WatchSource:0}: Error finding container 0e30ba34428ca4b575a16dd74ec9b7b1586b54393ffd8a48d4347031b7edda5d: Status 404 returned error can't find the container with id 0e30ba34428ca4b575a16dd74ec9b7b1586b54393ffd8a48d4347031b7edda5d Jul 10 19:06:36.075052 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:36.075014 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nz55p"] Jul 10 19:06:36.078504 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:36.078481 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nz55p" Jul 10 19:06:36.088053 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:36.088030 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nz55p"] Jul 10 19:06:36.103160 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:36.103128 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nz55p\" (UID: \"ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nz55p" Jul 10 19:06:36.103305 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:36.103183 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nz55p\" (UID: \"ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nz55p" Jul 10 19:06:36.103305 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:36.103261 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2snft\" (UniqueName: \"kubernetes.io/projected/ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc-kube-api-access-2snft\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nz55p\" (UID: \"ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nz55p" Jul 10 19:06:36.204701 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:36.204605 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nz55p\" (UID: \"ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nz55p" Jul 10 19:06:36.204701 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:36.204663 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nz55p\" (UID: \"ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nz55p" Jul 10 19:06:36.204887 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:36.204749 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2snft\" (UniqueName: \"kubernetes.io/projected/ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc-kube-api-access-2snft\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nz55p\" (UID: \"ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nz55p" Jul 10 19:06:36.205076 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:36.205053 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nz55p\" (UID: \"ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nz55p" Jul 10 19:06:36.205156 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:36.205034 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nz55p\" (UID: \"ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nz55p" Jul 10 19:06:36.213969 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:36.213937 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2snft\" (UniqueName: \"kubernetes.io/projected/ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc-kube-api-access-2snft\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nz55p\" (UID: \"ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nz55p" Jul 10 19:06:36.410450 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:36.410418 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nz55p" Jul 10 19:06:36.551822 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:36.551791 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nz55p"] Jul 10 19:06:36.554018 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:06:36.553987 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee1a3f74_1040_4f73_a7d5_904fd9b2c9bc.slice/crio-51c9d4e107427d57ce55e3f1429911a587249a3e6c7bc8bad892f8450754aa57 WatchSource:0}: Error finding container 51c9d4e107427d57ce55e3f1429911a587249a3e6c7bc8bad892f8450754aa57: Status 404 returned error can't find the container with id 51c9d4e107427d57ce55e3f1429911a587249a3e6c7bc8bad892f8450754aa57 Jul 10 19:06:36.720903 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:36.720816 2581 generic.go:356] "Generic (PLEG): container finished" podID="ec050446-3850-4b65-bcf8-2e7bf2688638" containerID="511e24d9cafb64691b885f46430fab7060f917fe029e9ab5788601ea9ed492f5" exitCode=0 Jul 10 19:06:36.721065 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:36.720906 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef156snh" event={"ID":"ec050446-3850-4b65-bcf8-2e7bf2688638","Type":"ContainerDied","Data":"511e24d9cafb64691b885f46430fab7060f917fe029e9ab5788601ea9ed492f5"} Jul 10 19:06:36.721065 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:36.720998 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef156snh" event={"ID":"ec050446-3850-4b65-bcf8-2e7bf2688638","Type":"ContainerStarted","Data":"0e30ba34428ca4b575a16dd74ec9b7b1586b54393ffd8a48d4347031b7edda5d"} Jul 10 19:06:36.722645 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:36.722620 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tx75x"] Jul 10 19:06:36.723511 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:36.723447 2581 generic.go:356] "Generic (PLEG): container finished" podID="8b0dd963-9313-419e-a83c-72908d524086" containerID="11098ff80128213046f1a7a69392b58119591c072c4342cdf81e882db128762d" exitCode=0 Jul 10 19:06:36.725411 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:36.725390 2581 generic.go:356] "Generic (PLEG): container finished" podID="ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc" containerID="c3279dfd59eac40685aad0b7d64fe2df9ca4827eea715d04718899d409ad4e3f" exitCode=0 Jul 10 19:06:36.726649 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:36.726628 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gx4nh" event={"ID":"8b0dd963-9313-419e-a83c-72908d524086","Type":"ContainerDied","Data":"11098ff80128213046f1a7a69392b58119591c072c4342cdf81e882db128762d"} Jul 10 19:06:36.726750 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:36.726669 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nz55p" event={"ID":"ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc","Type":"ContainerDied","Data":"c3279dfd59eac40685aad0b7d64fe2df9ca4827eea715d04718899d409ad4e3f"} Jul 10 19:06:36.726750 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:36.726712 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nz55p" event={"ID":"ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc","Type":"ContainerStarted","Data":"51c9d4e107427d57ce55e3f1429911a587249a3e6c7bc8bad892f8450754aa57"} Jul 10 19:06:36.726750 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:36.726737 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tx75x" Jul 10 19:06:36.734509 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:36.734289 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tx75x"] Jul 10 19:06:36.810053 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:36.810009 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/277dce86-11f7-4dc7-8209-c2021dfb69f2-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tx75x\" (UID: \"277dce86-11f7-4dc7-8209-c2021dfb69f2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tx75x" Jul 10 19:06:36.810234 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:36.810059 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/277dce86-11f7-4dc7-8209-c2021dfb69f2-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tx75x\" (UID: \"277dce86-11f7-4dc7-8209-c2021dfb69f2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tx75x" Jul 10 19:06:36.810234 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:36.810141 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjgl6\" (UniqueName: \"kubernetes.io/projected/277dce86-11f7-4dc7-8209-c2021dfb69f2-kube-api-access-tjgl6\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tx75x\" (UID: \"277dce86-11f7-4dc7-8209-c2021dfb69f2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tx75x" Jul 10 19:06:36.911324 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:36.911290 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/277dce86-11f7-4dc7-8209-c2021dfb69f2-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tx75x\" (UID: \"277dce86-11f7-4dc7-8209-c2021dfb69f2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tx75x" Jul 10 19:06:36.911324 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:36.911328 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/277dce86-11f7-4dc7-8209-c2021dfb69f2-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tx75x\" (UID: \"277dce86-11f7-4dc7-8209-c2021dfb69f2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tx75x" Jul 10 19:06:36.911559 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:36.911353 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjgl6\" (UniqueName: \"kubernetes.io/projected/277dce86-11f7-4dc7-8209-c2021dfb69f2-kube-api-access-tjgl6\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tx75x\" (UID: \"277dce86-11f7-4dc7-8209-c2021dfb69f2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tx75x" Jul 10 19:06:36.911722 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:36.911672 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/277dce86-11f7-4dc7-8209-c2021dfb69f2-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tx75x\" (UID: \"277dce86-11f7-4dc7-8209-c2021dfb69f2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tx75x" Jul 10 19:06:36.911799 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:36.911751 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/277dce86-11f7-4dc7-8209-c2021dfb69f2-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tx75x\" (UID: \"277dce86-11f7-4dc7-8209-c2021dfb69f2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tx75x" Jul 10 19:06:36.920340 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:36.920316 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjgl6\" (UniqueName: \"kubernetes.io/projected/277dce86-11f7-4dc7-8209-c2021dfb69f2-kube-api-access-tjgl6\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tx75x\" (UID: \"277dce86-11f7-4dc7-8209-c2021dfb69f2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tx75x" Jul 10 19:06:37.037092 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:37.037052 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tx75x" Jul 10 19:06:37.169156 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:37.169130 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tx75x"] Jul 10 19:06:37.169929 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:06:37.169907 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod277dce86_11f7_4dc7_8209_c2021dfb69f2.slice/crio-ca2bd81956b08a433f7b0dddbbb200444730d57d5abc67d2cc3da4130dc03659 WatchSource:0}: Error finding container ca2bd81956b08a433f7b0dddbbb200444730d57d5abc67d2cc3da4130dc03659: Status 404 returned error can't find the container with id ca2bd81956b08a433f7b0dddbbb200444730d57d5abc67d2cc3da4130dc03659 Jul 10 19:06:37.732007 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:37.731916 2581 generic.go:356] "Generic (PLEG): container finished" podID="ec050446-3850-4b65-bcf8-2e7bf2688638" containerID="f4126915b459f4f8e86e3386ca66b5f584eab4ce440a2e89363082434f5eb365" exitCode=0 Jul 10 19:06:37.732171 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:37.732001 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef156snh" event={"ID":"ec050446-3850-4b65-bcf8-2e7bf2688638","Type":"ContainerDied","Data":"f4126915b459f4f8e86e3386ca66b5f584eab4ce440a2e89363082434f5eb365"} Jul 10 19:06:37.734176 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:37.734150 2581 generic.go:356] "Generic (PLEG): container finished" podID="8b0dd963-9313-419e-a83c-72908d524086" containerID="8a9d7f087f5680c4e214dd5862b177f3b840e825f0579ee373a22cbfd0c24412" exitCode=0 Jul 10 19:06:37.734279 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:37.734232 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gx4nh" event={"ID":"8b0dd963-9313-419e-a83c-72908d524086","Type":"ContainerDied","Data":"8a9d7f087f5680c4e214dd5862b177f3b840e825f0579ee373a22cbfd0c24412"} Jul 10 19:06:37.735551 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:37.735532 2581 generic.go:356] "Generic (PLEG): container finished" podID="277dce86-11f7-4dc7-8209-c2021dfb69f2" containerID="63bc9989bad9c49c3b59963a5e5e34c9b6380ae504b4d6c50a595e8cfbb3b213" exitCode=0 Jul 10 19:06:37.735620 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:37.735589 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tx75x" event={"ID":"277dce86-11f7-4dc7-8209-c2021dfb69f2","Type":"ContainerDied","Data":"63bc9989bad9c49c3b59963a5e5e34c9b6380ae504b4d6c50a595e8cfbb3b213"} Jul 10 19:06:37.735664 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:37.735617 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tx75x" event={"ID":"277dce86-11f7-4dc7-8209-c2021dfb69f2","Type":"ContainerStarted","Data":"ca2bd81956b08a433f7b0dddbbb200444730d57d5abc67d2cc3da4130dc03659"} Jul 10 19:06:38.742308 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:38.742270 2581 generic.go:356] "Generic (PLEG): container finished" podID="ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc" containerID="f957b94d27d6d16cb40b22f16a2d5852f49554f22ba60df67aa91bcc9f7301aa" exitCode=0 Jul 10 19:06:38.742842 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:38.742322 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nz55p" event={"ID":"ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc","Type":"ContainerDied","Data":"f957b94d27d6d16cb40b22f16a2d5852f49554f22ba60df67aa91bcc9f7301aa"} Jul 10 19:06:38.744731 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:38.744664 2581 generic.go:356] "Generic (PLEG): container finished" podID="ec050446-3850-4b65-bcf8-2e7bf2688638" containerID="34306b130be87c177c24f594905f69537d6656e30b5009f0c083df9ce2142c30" exitCode=0 Jul 10 19:06:38.744731 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:38.744724 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef156snh" event={"ID":"ec050446-3850-4b65-bcf8-2e7bf2688638","Type":"ContainerDied","Data":"34306b130be87c177c24f594905f69537d6656e30b5009f0c083df9ce2142c30"} Jul 10 19:06:38.746458 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:38.746435 2581 generic.go:356] "Generic (PLEG): container finished" podID="277dce86-11f7-4dc7-8209-c2021dfb69f2" containerID="61522b4d2b22af85acf7b23af72edbdf6d6f7a51a012fc943d0290e70d219ba4" exitCode=0 Jul 10 19:06:38.746573 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:38.746509 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tx75x" event={"ID":"277dce86-11f7-4dc7-8209-c2021dfb69f2","Type":"ContainerDied","Data":"61522b4d2b22af85acf7b23af72edbdf6d6f7a51a012fc943d0290e70d219ba4"} Jul 10 19:06:38.885814 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:38.885784 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gx4nh" Jul 10 19:06:38.930088 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:38.930057 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8b0dd963-9313-419e-a83c-72908d524086-bundle\") pod \"8b0dd963-9313-419e-a83c-72908d524086\" (UID: \"8b0dd963-9313-419e-a83c-72908d524086\") " Jul 10 19:06:38.930238 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:38.930096 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jdz4l\" (UniqueName: \"kubernetes.io/projected/8b0dd963-9313-419e-a83c-72908d524086-kube-api-access-jdz4l\") pod \"8b0dd963-9313-419e-a83c-72908d524086\" (UID: \"8b0dd963-9313-419e-a83c-72908d524086\") " Jul 10 19:06:38.930238 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:38.930129 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8b0dd963-9313-419e-a83c-72908d524086-util\") pod \"8b0dd963-9313-419e-a83c-72908d524086\" (UID: \"8b0dd963-9313-419e-a83c-72908d524086\") " Jul 10 19:06:38.930650 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:38.930620 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b0dd963-9313-419e-a83c-72908d524086-bundle" (OuterVolumeSpecName: "bundle") pod "8b0dd963-9313-419e-a83c-72908d524086" (UID: "8b0dd963-9313-419e-a83c-72908d524086"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:06:38.932131 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:38.932109 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b0dd963-9313-419e-a83c-72908d524086-kube-api-access-jdz4l" (OuterVolumeSpecName: "kube-api-access-jdz4l") pod "8b0dd963-9313-419e-a83c-72908d524086" (UID: "8b0dd963-9313-419e-a83c-72908d524086"). InnerVolumeSpecName "kube-api-access-jdz4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:06:38.936071 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:38.936050 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b0dd963-9313-419e-a83c-72908d524086-util" (OuterVolumeSpecName: "util") pod "8b0dd963-9313-419e-a83c-72908d524086" (UID: "8b0dd963-9313-419e-a83c-72908d524086"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:06:39.031041 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:39.031009 2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8b0dd963-9313-419e-a83c-72908d524086-bundle\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:06:39.031041 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:39.031037 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdz4l\" (UniqueName: \"kubernetes.io/projected/8b0dd963-9313-419e-a83c-72908d524086-kube-api-access-jdz4l\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:06:39.031041 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:39.031048 2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8b0dd963-9313-419e-a83c-72908d524086-util\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:06:39.754129 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:39.754097 2581 generic.go:356] "Generic (PLEG): container finished" podID="ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc" containerID="8c85e4f7635080ab41ee04059d5a1bc89de71428a041ff18270f14d5dded0ebe" exitCode=0 Jul 10 19:06:39.754543 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:39.754183 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nz55p" event={"ID":"ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc","Type":"ContainerDied","Data":"8c85e4f7635080ab41ee04059d5a1bc89de71428a041ff18270f14d5dded0ebe"} Jul 10 19:06:39.755939 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:39.755920 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gx4nh" Jul 10 19:06:39.755939 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:39.755926 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gx4nh" event={"ID":"8b0dd963-9313-419e-a83c-72908d524086","Type":"ContainerDied","Data":"202b123f66bbbee2ea5dcf86fc5ce4b061a86ccb8178fa5d1aac78efbe418cef"} Jul 10 19:06:39.756075 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:39.755959 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="202b123f66bbbee2ea5dcf86fc5ce4b061a86ccb8178fa5d1aac78efbe418cef" Jul 10 19:06:39.758085 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:39.758060 2581 generic.go:356] "Generic (PLEG): container finished" podID="277dce86-11f7-4dc7-8209-c2021dfb69f2" containerID="ad49974ee76f812ae9b0b275d54c00892bcdab18161c2f30faaf076b6b564332" exitCode=0 Jul 10 19:06:39.758195 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:39.758147 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tx75x" event={"ID":"277dce86-11f7-4dc7-8209-c2021dfb69f2","Type":"ContainerDied","Data":"ad49974ee76f812ae9b0b275d54c00892bcdab18161c2f30faaf076b6b564332"} Jul 10 19:06:39.899773 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:39.899747 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef156snh" Jul 10 19:06:39.937903 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:39.937872 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec050446-3850-4b65-bcf8-2e7bf2688638-util\") pod \"ec050446-3850-4b65-bcf8-2e7bf2688638\" (UID: \"ec050446-3850-4b65-bcf8-2e7bf2688638\") " Jul 10 19:06:39.938073 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:39.937932 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dkhb4\" (UniqueName: \"kubernetes.io/projected/ec050446-3850-4b65-bcf8-2e7bf2688638-kube-api-access-dkhb4\") pod \"ec050446-3850-4b65-bcf8-2e7bf2688638\" (UID: \"ec050446-3850-4b65-bcf8-2e7bf2688638\") " Jul 10 19:06:39.938073 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:39.937954 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec050446-3850-4b65-bcf8-2e7bf2688638-bundle\") pod \"ec050446-3850-4b65-bcf8-2e7bf2688638\" (UID: \"ec050446-3850-4b65-bcf8-2e7bf2688638\") " Jul 10 19:06:39.938517 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:39.938489 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec050446-3850-4b65-bcf8-2e7bf2688638-bundle" (OuterVolumeSpecName: "bundle") pod "ec050446-3850-4b65-bcf8-2e7bf2688638" (UID: "ec050446-3850-4b65-bcf8-2e7bf2688638"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:06:39.940085 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:39.940063 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec050446-3850-4b65-bcf8-2e7bf2688638-kube-api-access-dkhb4" (OuterVolumeSpecName: "kube-api-access-dkhb4") pod "ec050446-3850-4b65-bcf8-2e7bf2688638" (UID: "ec050446-3850-4b65-bcf8-2e7bf2688638"). InnerVolumeSpecName "kube-api-access-dkhb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:06:39.942959 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:39.942926 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec050446-3850-4b65-bcf8-2e7bf2688638-util" (OuterVolumeSpecName: "util") pod "ec050446-3850-4b65-bcf8-2e7bf2688638" (UID: "ec050446-3850-4b65-bcf8-2e7bf2688638"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:06:40.038743 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:40.038639 2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec050446-3850-4b65-bcf8-2e7bf2688638-util\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:06:40.038743 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:40.038672 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkhb4\" (UniqueName: \"kubernetes.io/projected/ec050446-3850-4b65-bcf8-2e7bf2688638-kube-api-access-dkhb4\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:06:40.038743 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:40.038702 2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec050446-3850-4b65-bcf8-2e7bf2688638-bundle\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:06:40.764446 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:40.764417 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef156snh" Jul 10 19:06:40.764842 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:40.764443 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef156snh" event={"ID":"ec050446-3850-4b65-bcf8-2e7bf2688638","Type":"ContainerDied","Data":"0e30ba34428ca4b575a16dd74ec9b7b1586b54393ffd8a48d4347031b7edda5d"} Jul 10 19:06:40.764842 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:40.764500 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e30ba34428ca4b575a16dd74ec9b7b1586b54393ffd8a48d4347031b7edda5d" Jul 10 19:06:40.897307 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:40.897279 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tx75x" Jul 10 19:06:40.930609 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:40.930578 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nz55p" Jul 10 19:06:40.945274 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:40.945250 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjgl6\" (UniqueName: \"kubernetes.io/projected/277dce86-11f7-4dc7-8209-c2021dfb69f2-kube-api-access-tjgl6\") pod \"277dce86-11f7-4dc7-8209-c2021dfb69f2\" (UID: \"277dce86-11f7-4dc7-8209-c2021dfb69f2\") " Jul 10 19:06:40.945414 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:40.945287 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/277dce86-11f7-4dc7-8209-c2021dfb69f2-bundle\") pod \"277dce86-11f7-4dc7-8209-c2021dfb69f2\" (UID: \"277dce86-11f7-4dc7-8209-c2021dfb69f2\") " Jul 10 19:06:40.945414 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:40.945335 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/277dce86-11f7-4dc7-8209-c2021dfb69f2-util\") pod \"277dce86-11f7-4dc7-8209-c2021dfb69f2\" (UID: \"277dce86-11f7-4dc7-8209-c2021dfb69f2\") " Jul 10 19:06:40.945957 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:40.945932 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277dce86-11f7-4dc7-8209-c2021dfb69f2-bundle" (OuterVolumeSpecName: "bundle") pod "277dce86-11f7-4dc7-8209-c2021dfb69f2" (UID: "277dce86-11f7-4dc7-8209-c2021dfb69f2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:06:40.947411 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:40.947380 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/277dce86-11f7-4dc7-8209-c2021dfb69f2-kube-api-access-tjgl6" (OuterVolumeSpecName: "kube-api-access-tjgl6") pod "277dce86-11f7-4dc7-8209-c2021dfb69f2" (UID: "277dce86-11f7-4dc7-8209-c2021dfb69f2"). InnerVolumeSpecName "kube-api-access-tjgl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:06:40.950745 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:40.950725 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277dce86-11f7-4dc7-8209-c2021dfb69f2-util" (OuterVolumeSpecName: "util") pod "277dce86-11f7-4dc7-8209-c2021dfb69f2" (UID: "277dce86-11f7-4dc7-8209-c2021dfb69f2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:06:41.046615 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:41.046583 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc-bundle\") pod \"ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc\" (UID: \"ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc\") " Jul 10 19:06:41.046768 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:41.046661 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc-util\") pod \"ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc\" (UID: \"ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc\") " Jul 10 19:06:41.046827 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:41.046765 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2snft\" (UniqueName: \"kubernetes.io/projected/ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc-kube-api-access-2snft\") pod \"ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc\" (UID: \"ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc\") " Jul 10 19:06:41.047033 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:41.047015 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjgl6\" (UniqueName: \"kubernetes.io/projected/277dce86-11f7-4dc7-8209-c2021dfb69f2-kube-api-access-tjgl6\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:06:41.047073 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:41.047040 2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/277dce86-11f7-4dc7-8209-c2021dfb69f2-bundle\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:06:41.047073 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:41.047055 2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/277dce86-11f7-4dc7-8209-c2021dfb69f2-util\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:06:41.047207 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:41.047184 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc-bundle" (OuterVolumeSpecName: "bundle") pod "ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc" (UID: "ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:06:41.048831 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:41.048807 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc-kube-api-access-2snft" (OuterVolumeSpecName: "kube-api-access-2snft") pod "ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc" (UID: "ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc"). InnerVolumeSpecName "kube-api-access-2snft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:06:41.052297 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:41.052275 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc-util" (OuterVolumeSpecName: "util") pod "ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc" (UID: "ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:06:41.147848 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:41.147753 2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc-bundle\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:06:41.147848 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:41.147787 2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc-util\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:06:41.147848 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:41.147798 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2snft\" (UniqueName: \"kubernetes.io/projected/ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc-kube-api-access-2snft\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:06:41.770999 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:41.770966 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tx75x" Jul 10 19:06:41.771515 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:41.770996 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tx75x" event={"ID":"277dce86-11f7-4dc7-8209-c2021dfb69f2","Type":"ContainerDied","Data":"ca2bd81956b08a433f7b0dddbbb200444730d57d5abc67d2cc3da4130dc03659"} Jul 10 19:06:41.771515 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:41.771075 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca2bd81956b08a433f7b0dddbbb200444730d57d5abc67d2cc3da4130dc03659" Jul 10 19:06:41.772860 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:41.772826 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nz55p" event={"ID":"ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc","Type":"ContainerDied","Data":"51c9d4e107427d57ce55e3f1429911a587249a3e6c7bc8bad892f8450754aa57"} Jul 10 19:06:41.772969 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:41.772860 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nz55p" Jul 10 19:06:41.772969 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:41.772869 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51c9d4e107427d57ce55e3f1429911a587249a3e6c7bc8bad892f8450754aa57" Jul 10 19:06:45.650595 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:45.650559 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jdjt9"] Jul 10 19:06:45.650994 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:45.650968 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec050446-3850-4b65-bcf8-2e7bf2688638" containerName="pull" Jul 10 19:06:45.650994 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:45.650983 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec050446-3850-4b65-bcf8-2e7bf2688638" containerName="pull" Jul 10 19:06:45.650994 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:45.650993 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b0dd963-9313-419e-a83c-72908d524086" containerName="pull" Jul 10 19:06:45.651115 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:45.650999 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b0dd963-9313-419e-a83c-72908d524086" containerName="pull" Jul 10 19:06:45.651115 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:45.651006 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc" containerName="pull" Jul 10 19:06:45.651115 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:45.651012 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc" containerName="pull" Jul 10 19:06:45.651115 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:45.651019 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="277dce86-11f7-4dc7-8209-c2021dfb69f2" containerName="extract" Jul 10 19:06:45.651115 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:45.651026 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="277dce86-11f7-4dc7-8209-c2021dfb69f2" containerName="extract" Jul 10 19:06:45.651115 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:45.651041 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="277dce86-11f7-4dc7-8209-c2021dfb69f2" containerName="util" Jul 10 19:06:45.651115 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:45.651050 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="277dce86-11f7-4dc7-8209-c2021dfb69f2" containerName="util" Jul 10 19:06:45.651115 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:45.651058 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc" containerName="util" Jul 10 19:06:45.651115 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:45.651063 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc" containerName="util" Jul 10 19:06:45.651115 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:45.651070 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b0dd963-9313-419e-a83c-72908d524086" containerName="util" Jul 10 19:06:45.651115 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:45.651074 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b0dd963-9313-419e-a83c-72908d524086" containerName="util" Jul 10 19:06:45.651115 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:45.651080 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec050446-3850-4b65-bcf8-2e7bf2688638" containerName="util" Jul 10 19:06:45.651115 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:45.651086 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec050446-3850-4b65-bcf8-2e7bf2688638" containerName="util" Jul 10 19:06:45.651115 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:45.651092 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b0dd963-9313-419e-a83c-72908d524086" containerName="extract" Jul 10 19:06:45.651115 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:45.651097 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b0dd963-9313-419e-a83c-72908d524086" containerName="extract" Jul 10 19:06:45.651115 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:45.651104 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc" containerName="extract" Jul 10 19:06:45.651115 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:45.651112 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc" containerName="extract" Jul 10 19:06:45.651592 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:45.651123 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="277dce86-11f7-4dc7-8209-c2021dfb69f2" containerName="pull" Jul 10 19:06:45.651592 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:45.651129 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="277dce86-11f7-4dc7-8209-c2021dfb69f2" containerName="pull" Jul 10 19:06:45.651592 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:45.651137 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec050446-3850-4b65-bcf8-2e7bf2688638" containerName="extract" Jul 10 19:06:45.651592 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:45.651142 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec050446-3850-4b65-bcf8-2e7bf2688638" containerName="extract" Jul 10 19:06:45.651592 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:45.651194 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc" containerName="extract" Jul 10 19:06:45.651592 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:45.651207 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec050446-3850-4b65-bcf8-2e7bf2688638" containerName="extract" Jul 10 19:06:45.651592 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:45.651213 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b0dd963-9313-419e-a83c-72908d524086" containerName="extract" Jul 10 19:06:45.651592 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:45.651219 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="277dce86-11f7-4dc7-8209-c2021dfb69f2" containerName="extract" Jul 10 19:06:45.655521 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:45.655503 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jdjt9" Jul 10 19:06:45.663061 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:45.663044 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-dkvkv\"" Jul 10 19:06:45.671632 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:45.671601 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jdjt9"] Jul 10 19:06:45.689126 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:45.689101 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6vg6\" (UniqueName: \"kubernetes.io/projected/e2cc3070-ce84-451b-9954-cea775baffdc-kube-api-access-j6vg6\") pod \"limitador-operator-controller-manager-859d7b78bd-jdjt9\" (UID: \"e2cc3070-ce84-451b-9954-cea775baffdc\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jdjt9" Jul 10 19:06:45.789514 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:45.789481 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6vg6\" (UniqueName: \"kubernetes.io/projected/e2cc3070-ce84-451b-9954-cea775baffdc-kube-api-access-j6vg6\") pod \"limitador-operator-controller-manager-859d7b78bd-jdjt9\" (UID: \"e2cc3070-ce84-451b-9954-cea775baffdc\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jdjt9" Jul 10 19:06:45.800257 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:45.800221 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6vg6\" (UniqueName: \"kubernetes.io/projected/e2cc3070-ce84-451b-9954-cea775baffdc-kube-api-access-j6vg6\") pod \"limitador-operator-controller-manager-859d7b78bd-jdjt9\" (UID: \"e2cc3070-ce84-451b-9954-cea775baffdc\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jdjt9" Jul 10 19:06:45.965804 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:45.965710 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jdjt9" Jul 10 19:06:46.104375 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:46.104349 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jdjt9"] Jul 10 19:06:46.106640 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:06:46.106611 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2cc3070_ce84_451b_9954_cea775baffdc.slice/crio-872f3b09caabf0d7b5975dfc52ca8aea378fa3656847f81e4a5cd1fab1cdf04d WatchSource:0}: Error finding container 872f3b09caabf0d7b5975dfc52ca8aea378fa3656847f81e4a5cd1fab1cdf04d: Status 404 returned error can't find the container with id 872f3b09caabf0d7b5975dfc52ca8aea378fa3656847f81e4a5cd1fab1cdf04d Jul 10 19:06:46.802609 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:46.802578 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jdjt9" event={"ID":"e2cc3070-ce84-451b-9954-cea775baffdc","Type":"ContainerStarted","Data":"872f3b09caabf0d7b5975dfc52ca8aea378fa3656847f81e4a5cd1fab1cdf04d"} Jul 10 19:06:48.813037 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:48.813001 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jdjt9" event={"ID":"e2cc3070-ce84-451b-9954-cea775baffdc","Type":"ContainerStarted","Data":"532a012ad219b37f817d2c146b06f6b0b9d73f70f8f53399f5dd620637d3c44b"} Jul 10 19:06:48.813471 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:48.813062 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jdjt9" Jul 10 19:06:48.851808 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:48.851758 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jdjt9" podStartSLOduration=2.158782467 podStartE2EDuration="3.851742363s" totalImagesPullingTime="1.692959896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:06:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:06:48.8466687 +0000 UTC m=+492.890811214" watchObservedRunningTime="2026-07-10 19:06:48.851742363 +0000 UTC m=+492.895884867" Jul 10 19:06:51.006431 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:51.006397 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dvj48"] Jul 10 19:06:51.009867 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:51.009845 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dvj48" Jul 10 19:06:51.013270 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:51.013248 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-lt9xj\"" Jul 10 19:06:51.025729 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:51.025671 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dvj48"] Jul 10 19:06:51.136641 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:51.136601 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x2cj\" (UniqueName: \"kubernetes.io/projected/85301a7d-a72b-4b92-b266-176eb0c1625f-kube-api-access-2x2cj\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dvj48\" (UID: \"85301a7d-a72b-4b92-b266-176eb0c1625f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dvj48" Jul 10 19:06:51.136842 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:51.136658 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85301a7d-a72b-4b92-b266-176eb0c1625f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dvj48\" (UID: \"85301a7d-a72b-4b92-b266-176eb0c1625f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dvj48" Jul 10 19:06:51.237176 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:51.237139 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85301a7d-a72b-4b92-b266-176eb0c1625f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dvj48\" (UID: \"85301a7d-a72b-4b92-b266-176eb0c1625f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dvj48" Jul 10 19:06:51.237353 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:51.237196 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x2cj\" (UniqueName: \"kubernetes.io/projected/85301a7d-a72b-4b92-b266-176eb0c1625f-kube-api-access-2x2cj\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dvj48\" (UID: \"85301a7d-a72b-4b92-b266-176eb0c1625f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dvj48" Jul 10 19:06:51.237527 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:51.237505 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85301a7d-a72b-4b92-b266-176eb0c1625f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dvj48\" (UID: \"85301a7d-a72b-4b92-b266-176eb0c1625f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dvj48" Jul 10 19:06:51.255426 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:51.255384 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x2cj\" (UniqueName: \"kubernetes.io/projected/85301a7d-a72b-4b92-b266-176eb0c1625f-kube-api-access-2x2cj\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dvj48\" (UID: \"85301a7d-a72b-4b92-b266-176eb0c1625f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dvj48" Jul 10 19:06:51.320149 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:51.320057 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dvj48" Jul 10 19:06:51.458610 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:51.458586 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dvj48"] Jul 10 19:06:51.460525 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:06:51.460487 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85301a7d_a72b_4b92_b266_176eb0c1625f.slice/crio-2bac4a7ad2e802531f57b52bbd6b8014c2d1ba837c5834cb336c872baf922d95 WatchSource:0}: Error finding container 2bac4a7ad2e802531f57b52bbd6b8014c2d1ba837c5834cb336c872baf922d95: Status 404 returned error can't find the container with id 2bac4a7ad2e802531f57b52bbd6b8014c2d1ba837c5834cb336c872baf922d95 Jul 10 19:06:51.827642 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:51.827604 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dvj48" event={"ID":"85301a7d-a72b-4b92-b266-176eb0c1625f","Type":"ContainerStarted","Data":"2bac4a7ad2e802531f57b52bbd6b8014c2d1ba837c5834cb336c872baf922d95"} Jul 10 19:06:54.291938 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:54.291896 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-bs99p"] Jul 10 19:06:54.295776 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:54.295753 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bs99p" Jul 10 19:06:54.299558 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:54.299532 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 10 19:06:54.299709 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:54.299581 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-x6p6f\"" Jul 10 19:06:54.319771 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:54.319740 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-bs99p"] Jul 10 19:06:54.361382 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:54.361349 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlkvs\" (UniqueName: \"kubernetes.io/projected/f30bbd65-1ec6-44df-80ff-15b3e3e32b7b-kube-api-access-xlkvs\") pod \"dns-operator-controller-manager-864dd78c4c-bs99p\" (UID: \"f30bbd65-1ec6-44df-80ff-15b3e3e32b7b\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bs99p" Jul 10 19:06:54.461818 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:54.461779 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlkvs\" (UniqueName: \"kubernetes.io/projected/f30bbd65-1ec6-44df-80ff-15b3e3e32b7b-kube-api-access-xlkvs\") pod \"dns-operator-controller-manager-864dd78c4c-bs99p\" (UID: \"f30bbd65-1ec6-44df-80ff-15b3e3e32b7b\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bs99p" Jul 10 19:06:54.473924 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:54.473886 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlkvs\" (UniqueName: \"kubernetes.io/projected/f30bbd65-1ec6-44df-80ff-15b3e3e32b7b-kube-api-access-xlkvs\") pod \"dns-operator-controller-manager-864dd78c4c-bs99p\" (UID: \"f30bbd65-1ec6-44df-80ff-15b3e3e32b7b\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bs99p" Jul 10 19:06:54.610002 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:54.609914 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bs99p" Jul 10 19:06:54.760035 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:54.760003 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-bs99p"] Jul 10 19:06:54.760710 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:06:54.760657 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf30bbd65_1ec6_44df_80ff_15b3e3e32b7b.slice/crio-43fb2bf57bad30323ecba3ef0e5d446aaea8b4d01bac88a53575c3eaa0f9dfce WatchSource:0}: Error finding container 43fb2bf57bad30323ecba3ef0e5d446aaea8b4d01bac88a53575c3eaa0f9dfce: Status 404 returned error can't find the container with id 43fb2bf57bad30323ecba3ef0e5d446aaea8b4d01bac88a53575c3eaa0f9dfce Jul 10 19:06:54.844632 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:54.844585 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bs99p" event={"ID":"f30bbd65-1ec6-44df-80ff-15b3e3e32b7b","Type":"ContainerStarted","Data":"43fb2bf57bad30323ecba3ef0e5d446aaea8b4d01bac88a53575c3eaa0f9dfce"} Jul 10 19:06:57.756544 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:57.756512 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-hmfrt"] Jul 10 19:06:57.758931 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:57.758913 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hmfrt" Jul 10 19:06:57.761596 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:57.761577 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-8jjgj\"" Jul 10 19:06:57.769607 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:57.769580 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-hmfrt"] Jul 10 19:06:57.865662 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:57.865559 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dvj48" event={"ID":"85301a7d-a72b-4b92-b266-176eb0c1625f","Type":"ContainerStarted","Data":"a0952420c607c5b63a69154f5bb9663646de747efb6e1b0c30736383ca1aad9d"} Jul 10 19:06:57.865826 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:57.865727 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dvj48" Jul 10 19:06:57.867171 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:57.867145 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bs99p" event={"ID":"f30bbd65-1ec6-44df-80ff-15b3e3e32b7b","Type":"ContainerStarted","Data":"fd525dd1e33509a713091079343b82cfc2adf81294444e5d669d9d143c72cecf"} Jul 10 19:06:57.867280 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:57.867263 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bs99p" Jul 10 19:06:57.885519 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:57.885474 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dvj48" podStartSLOduration=1.72414553 podStartE2EDuration="7.885459204s" totalImagesPullingTime="6.161313674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:06:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:06:57.883208522 +0000 UTC m=+501.927351068" watchObservedRunningTime="2026-07-10 19:06:57.885459204 +0000 UTC m=+501.929601705" Jul 10 19:06:57.894481 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:57.894455 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dltw6\" (UniqueName: \"kubernetes.io/projected/4df8aa21-bb25-467b-b0c2-6392ff510974-kube-api-access-dltw6\") pod \"authorino-operator-75cc8d5cb7-hmfrt\" (UID: \"4df8aa21-bb25-467b-b0c2-6392ff510974\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-hmfrt" Jul 10 19:06:57.907481 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:57.907435 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bs99p" podStartSLOduration=1.056676 podStartE2EDuration="3.907419184s" totalImagesPullingTime="2.850743184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:06:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:06:57.905080776 +0000 UTC m=+501.949223296" watchObservedRunningTime="2026-07-10 19:06:57.907419184 +0000 UTC m=+501.951561685" Jul 10 19:06:57.995639 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:57.995600 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dltw6\" (UniqueName: \"kubernetes.io/projected/4df8aa21-bb25-467b-b0c2-6392ff510974-kube-api-access-dltw6\") pod \"authorino-operator-75cc8d5cb7-hmfrt\" (UID: \"4df8aa21-bb25-467b-b0c2-6392ff510974\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-hmfrt" Jul 10 19:06:58.012644 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:58.012613 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dltw6\" (UniqueName: \"kubernetes.io/projected/4df8aa21-bb25-467b-b0c2-6392ff510974-kube-api-access-dltw6\") pod \"authorino-operator-75cc8d5cb7-hmfrt\" (UID: \"4df8aa21-bb25-467b-b0c2-6392ff510974\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-hmfrt" Jul 10 19:06:58.069409 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:58.069370 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hmfrt" Jul 10 19:06:58.212066 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:58.212038 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-hmfrt"] Jul 10 19:06:58.214551 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:06:58.214525 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4df8aa21_bb25_467b_b0c2_6392ff510974.slice/crio-87ac85ca7762653d31afacfaf7def91b0d1b28f3f6a3ce14736a817442793d50 WatchSource:0}: Error finding container 87ac85ca7762653d31afacfaf7def91b0d1b28f3f6a3ce14736a817442793d50: Status 404 returned error can't find the container with id 87ac85ca7762653d31afacfaf7def91b0d1b28f3f6a3ce14736a817442793d50 Jul 10 19:06:58.874699 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:58.874640 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hmfrt" event={"ID":"4df8aa21-bb25-467b-b0c2-6392ff510974","Type":"ContainerStarted","Data":"87ac85ca7762653d31afacfaf7def91b0d1b28f3f6a3ce14736a817442793d50"} Jul 10 19:06:59.821315 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:06:59.821284 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jdjt9" Jul 10 19:07:00.824514 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:00.824472 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-qtvk4"] Jul 10 19:07:00.828011 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:00.827988 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qtvk4" Jul 10 19:07:00.830693 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:00.830655 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 10 19:07:00.830813 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:00.830656 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-4d9qg\"" Jul 10 19:07:00.830813 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:00.830679 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 10 19:07:00.837465 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:00.837438 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-qtvk4"] Jul 10 19:07:00.886286 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:00.886248 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hmfrt" event={"ID":"4df8aa21-bb25-467b-b0c2-6392ff510974","Type":"ContainerStarted","Data":"9ca22ef5cea104fd670a708af97b82febae8c040711fdebab377e0b3cd131010"} Jul 10 19:07:00.886460 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:00.886403 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hmfrt" Jul 10 19:07:00.903954 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:00.903906 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hmfrt" podStartSLOduration=1.828915268 podStartE2EDuration="3.90389158s" totalImagesPullingTime="2.074976312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:06:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:07:00.902197417 +0000 UTC m=+504.946339920" watchObservedRunningTime="2026-07-10 19:07:00.90389158 +0000 UTC m=+504.948034081" Jul 10 19:07:00.923928 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:00.923897 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b260e5a5-56f3-47eb-9875-c2d0f740782a-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-qtvk4\" (UID: \"b260e5a5-56f3-47eb-9875-c2d0f740782a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qtvk4" Jul 10 19:07:00.924067 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:00.923932 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4655p\" (UniqueName: \"kubernetes.io/projected/b260e5a5-56f3-47eb-9875-c2d0f740782a-kube-api-access-4655p\") pod \"kuadrant-console-plugin-66b5774985-qtvk4\" (UID: \"b260e5a5-56f3-47eb-9875-c2d0f740782a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qtvk4" Jul 10 19:07:00.924067 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:00.923955 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b260e5a5-56f3-47eb-9875-c2d0f740782a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-qtvk4\" (UID: \"b260e5a5-56f3-47eb-9875-c2d0f740782a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qtvk4" Jul 10 19:07:01.025023 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:01.024989 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b260e5a5-56f3-47eb-9875-c2d0f740782a-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-qtvk4\" (UID: \"b260e5a5-56f3-47eb-9875-c2d0f740782a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qtvk4" Jul 10 19:07:01.025200 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:01.025036 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4655p\" (UniqueName: \"kubernetes.io/projected/b260e5a5-56f3-47eb-9875-c2d0f740782a-kube-api-access-4655p\") pod \"kuadrant-console-plugin-66b5774985-qtvk4\" (UID: \"b260e5a5-56f3-47eb-9875-c2d0f740782a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qtvk4" Jul 10 19:07:01.025200 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:01.025071 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b260e5a5-56f3-47eb-9875-c2d0f740782a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-qtvk4\" (UID: \"b260e5a5-56f3-47eb-9875-c2d0f740782a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qtvk4" Jul 10 19:07:01.025882 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:01.025847 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b260e5a5-56f3-47eb-9875-c2d0f740782a-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-qtvk4\" (UID: \"b260e5a5-56f3-47eb-9875-c2d0f740782a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qtvk4" Jul 10 19:07:01.027507 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:01.027486 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b260e5a5-56f3-47eb-9875-c2d0f740782a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-qtvk4\" (UID: \"b260e5a5-56f3-47eb-9875-c2d0f740782a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qtvk4" Jul 10 19:07:01.039489 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:01.039463 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4655p\" (UniqueName: \"kubernetes.io/projected/b260e5a5-56f3-47eb-9875-c2d0f740782a-kube-api-access-4655p\") pod \"kuadrant-console-plugin-66b5774985-qtvk4\" (UID: \"b260e5a5-56f3-47eb-9875-c2d0f740782a\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qtvk4" Jul 10 19:07:01.138074 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:01.137983 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qtvk4" Jul 10 19:07:01.279723 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:01.277532 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-qtvk4"] Jul 10 19:07:01.892367 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:01.892324 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qtvk4" event={"ID":"b260e5a5-56f3-47eb-9875-c2d0f740782a","Type":"ContainerStarted","Data":"885d6f999d6088118b5ac70f27fd9688e2320968837f0c1ecbab6cd9fcac40c3"} Jul 10 19:07:08.877089 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:08.877057 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dvj48" Jul 10 19:07:08.877459 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:08.877129 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bs99p" Jul 10 19:07:10.721574 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:10.721515 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dvj48"] Jul 10 19:07:10.722223 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:10.721763 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dvj48" podUID="85301a7d-a72b-4b92-b266-176eb0c1625f" containerName="manager" containerID="cri-o://a0952420c607c5b63a69154f5bb9663646de747efb6e1b0c30736383ca1aad9d" gracePeriod=2 Jul 10 19:07:10.724788 ip-10-0-135-67 kubenswrapper[2581]: E0710 19:07:10.724742 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dvj48\" is forbidden: User \"system:node:ip-10-0-135-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-67.ec2.internal' and this object" podUID="85301a7d-a72b-4b92-b266-176eb0c1625f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dvj48" Jul 10 19:07:10.725338 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:10.725308 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dvj48"] Jul 10 19:07:10.744962 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:10.744932 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jdjt9"] Jul 10 19:07:10.745238 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:10.745188 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jdjt9" podUID="e2cc3070-ce84-451b-9954-cea775baffdc" containerName="manager" containerID="cri-o://532a012ad219b37f817d2c146b06f6b0b9d73f70f8f53399f5dd620637d3c44b" gracePeriod=2 Jul 10 19:07:10.748452 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:10.748378 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-j7rlm"] Jul 10 19:07:10.749164 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:10.749143 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85301a7d-a72b-4b92-b266-176eb0c1625f" containerName="manager" Jul 10 19:07:10.749383 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:10.749349 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="85301a7d-a72b-4b92-b266-176eb0c1625f" containerName="manager" Jul 10 19:07:10.750600 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:10.750505 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="85301a7d-a72b-4b92-b266-176eb0c1625f" containerName="manager" Jul 10 19:07:10.755217 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:10.755181 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jdjt9"] Jul 10 19:07:10.755337 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:10.755316 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-j7rlm" Jul 10 19:07:10.758468 ip-10-0-135-67 kubenswrapper[2581]: E0710 19:07:10.758431 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dvj48\" is forbidden: User \"system:node:ip-10-0-135-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-67.ec2.internal' and this object" podUID="85301a7d-a72b-4b92-b266-176eb0c1625f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dvj48" Jul 10 19:07:10.765828 ip-10-0-135-67 kubenswrapper[2581]: E0710 19:07:10.765798 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jdjt9\" is forbidden: User \"system:node:ip-10-0-135-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-67.ec2.internal' and this object" podUID="e2cc3070-ce84-451b-9954-cea775baffdc" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jdjt9" Jul 10 19:07:10.768872 ip-10-0-135-67 kubenswrapper[2581]: E0710 19:07:10.768829 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dvj48\" is forbidden: User \"system:node:ip-10-0-135-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-67.ec2.internal' and this object" podUID="85301a7d-a72b-4b92-b266-176eb0c1625f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dvj48" Jul 10 19:07:10.785338 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:10.785287 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kr8fm"] Jul 10 19:07:10.785865 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:10.785846 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2cc3070-ce84-451b-9954-cea775baffdc" containerName="manager" Jul 10 19:07:10.786005 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:10.785994 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2cc3070-ce84-451b-9954-cea775baffdc" containerName="manager" Jul 10 19:07:10.786181 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:10.786170 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2cc3070-ce84-451b-9954-cea775baffdc" containerName="manager" Jul 10 19:07:10.791196 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:10.791171 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-j7rlm"] Jul 10 19:07:10.791524 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:10.791376 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kr8fm" Jul 10 19:07:10.801194 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:10.801165 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kr8fm"] Jul 10 19:07:10.805856 ip-10-0-135-67 kubenswrapper[2581]: E0710 19:07:10.805822 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jdjt9\" is forbidden: User \"system:node:ip-10-0-135-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-67.ec2.internal' and this object" podUID="e2cc3070-ce84-451b-9954-cea775baffdc" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jdjt9" Jul 10 19:07:10.807880 ip-10-0-135-67 kubenswrapper[2581]: E0710 19:07:10.807841 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dvj48\" is forbidden: User \"system:node:ip-10-0-135-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-67.ec2.internal' and this object" podUID="85301a7d-a72b-4b92-b266-176eb0c1625f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dvj48" Jul 10 19:07:10.826948 ip-10-0-135-67 kubenswrapper[2581]: E0710 19:07:10.826903 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jdjt9\" is forbidden: User \"system:node:ip-10-0-135-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-67.ec2.internal' and this object" podUID="e2cc3070-ce84-451b-9954-cea775baffdc" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jdjt9" Jul 10 19:07:10.919592 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:10.919555 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmlr4\" (UniqueName: \"kubernetes.io/projected/18f05682-360f-40e3-a82a-b94fdb255a5a-kube-api-access-tmlr4\") pod \"kuadrant-operator-controller-manager-8684c9ff49-j7rlm\" (UID: \"18f05682-360f-40e3-a82a-b94fdb255a5a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-j7rlm" Jul 10 19:07:10.919592 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:10.919600 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/18f05682-360f-40e3-a82a-b94fdb255a5a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-j7rlm\" (UID: \"18f05682-360f-40e3-a82a-b94fdb255a5a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-j7rlm" Jul 10 19:07:10.919833 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:10.919671 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjzsq\" (UniqueName: \"kubernetes.io/projected/3c6823d8-1cae-4102-89db-5c7d8e30d7aa-kube-api-access-bjzsq\") pod \"limitador-operator-controller-manager-859d7b78bd-kr8fm\" (UID: \"3c6823d8-1cae-4102-89db-5c7d8e30d7aa\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kr8fm" Jul 10 19:07:11.020418 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:11.020380 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjzsq\" (UniqueName: \"kubernetes.io/projected/3c6823d8-1cae-4102-89db-5c7d8e30d7aa-kube-api-access-bjzsq\") pod \"limitador-operator-controller-manager-859d7b78bd-kr8fm\" (UID: \"3c6823d8-1cae-4102-89db-5c7d8e30d7aa\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kr8fm" Jul 10 19:07:11.020599 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:11.020482 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmlr4\" (UniqueName: \"kubernetes.io/projected/18f05682-360f-40e3-a82a-b94fdb255a5a-kube-api-access-tmlr4\") pod \"kuadrant-operator-controller-manager-8684c9ff49-j7rlm\" (UID: \"18f05682-360f-40e3-a82a-b94fdb255a5a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-j7rlm" Jul 10 19:07:11.020599 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:11.020510 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/18f05682-360f-40e3-a82a-b94fdb255a5a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-j7rlm\" (UID: \"18f05682-360f-40e3-a82a-b94fdb255a5a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-j7rlm" Jul 10 19:07:11.020959 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:11.020938 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/18f05682-360f-40e3-a82a-b94fdb255a5a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-j7rlm\" (UID: \"18f05682-360f-40e3-a82a-b94fdb255a5a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-j7rlm" Jul 10 19:07:11.030446 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:11.030413 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjzsq\" (UniqueName: \"kubernetes.io/projected/3c6823d8-1cae-4102-89db-5c7d8e30d7aa-kube-api-access-bjzsq\") pod \"limitador-operator-controller-manager-859d7b78bd-kr8fm\" (UID: \"3c6823d8-1cae-4102-89db-5c7d8e30d7aa\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kr8fm" Jul 10 19:07:11.031072 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:11.031045 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmlr4\" (UniqueName: \"kubernetes.io/projected/18f05682-360f-40e3-a82a-b94fdb255a5a-kube-api-access-tmlr4\") pod \"kuadrant-operator-controller-manager-8684c9ff49-j7rlm\" (UID: \"18f05682-360f-40e3-a82a-b94fdb255a5a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-j7rlm" Jul 10 19:07:11.129480 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:11.129438 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-j7rlm" Jul 10 19:07:11.138324 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:11.138298 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kr8fm" Jul 10 19:07:11.895184 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:11.895149 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hmfrt" Jul 10 19:07:11.897556 ip-10-0-135-67 kubenswrapper[2581]: E0710 19:07:11.897518 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dvj48\" is forbidden: User \"system:node:ip-10-0-135-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-67.ec2.internal' and this object" podUID="85301a7d-a72b-4b92-b266-176eb0c1625f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dvj48" Jul 10 19:07:11.925570 ip-10-0-135-67 kubenswrapper[2581]: E0710 19:07:11.925527 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jdjt9\" is forbidden: User \"system:node:ip-10-0-135-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-67.ec2.internal' and this object" podUID="e2cc3070-ce84-451b-9954-cea775baffdc" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jdjt9" Jul 10 19:07:12.184257 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:12.184173 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wqfnm"] Jul 10 19:07:12.203882 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:12.203815 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wqfnm"] Jul 10 19:07:12.204073 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:12.203935 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wqfnm" Jul 10 19:07:12.206368 ip-10-0-135-67 kubenswrapper[2581]: E0710 19:07:12.206328 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dvj48\" is forbidden: User \"system:node:ip-10-0-135-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-67.ec2.internal' and this object" podUID="85301a7d-a72b-4b92-b266-176eb0c1625f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dvj48" Jul 10 19:07:12.225263 ip-10-0-135-67 kubenswrapper[2581]: E0710 19:07:12.225224 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jdjt9\" is forbidden: User \"system:node:ip-10-0-135-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-67.ec2.internal' and this object" podUID="e2cc3070-ce84-451b-9954-cea775baffdc" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jdjt9" Jul 10 19:07:12.333832 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:12.333777 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t29wp\" (UniqueName: \"kubernetes.io/projected/d7981090-2b0a-4f1f-949b-218b720d5f7c-kube-api-access-t29wp\") pod \"kuadrant-operator-controller-manager-86878c769f-wqfnm\" (UID: \"d7981090-2b0a-4f1f-949b-218b720d5f7c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wqfnm" Jul 10 19:07:12.334029 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:12.333850 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d7981090-2b0a-4f1f-949b-218b720d5f7c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-wqfnm\" (UID: \"d7981090-2b0a-4f1f-949b-218b720d5f7c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wqfnm" Jul 10 19:07:12.434669 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:12.434581 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t29wp\" (UniqueName: \"kubernetes.io/projected/d7981090-2b0a-4f1f-949b-218b720d5f7c-kube-api-access-t29wp\") pod \"kuadrant-operator-controller-manager-86878c769f-wqfnm\" (UID: \"d7981090-2b0a-4f1f-949b-218b720d5f7c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wqfnm" Jul 10 19:07:12.434669 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:12.434627 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d7981090-2b0a-4f1f-949b-218b720d5f7c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-wqfnm\" (UID: \"d7981090-2b0a-4f1f-949b-218b720d5f7c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wqfnm" Jul 10 19:07:12.435034 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:12.435015 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d7981090-2b0a-4f1f-949b-218b720d5f7c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-wqfnm\" (UID: \"d7981090-2b0a-4f1f-949b-218b720d5f7c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wqfnm" Jul 10 19:07:12.451189 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:12.451155 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t29wp\" (UniqueName: \"kubernetes.io/projected/d7981090-2b0a-4f1f-949b-218b720d5f7c-kube-api-access-t29wp\") pod \"kuadrant-operator-controller-manager-86878c769f-wqfnm\" (UID: \"d7981090-2b0a-4f1f-949b-218b720d5f7c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wqfnm" Jul 10 19:07:12.517403 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:12.517359 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wqfnm" Jul 10 19:07:24.640288 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:24.640132 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dvj48" Jul 10 19:07:24.642967 ip-10-0-135-67 kubenswrapper[2581]: E0710 19:07:24.642919 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dvj48\" is forbidden: User \"system:node:ip-10-0-135-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-67.ec2.internal' and this object" podUID="85301a7d-a72b-4b92-b266-176eb0c1625f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dvj48" Jul 10 19:07:24.644630 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:24.644601 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jdjt9" Jul 10 19:07:24.647963 ip-10-0-135-67 kubenswrapper[2581]: E0710 19:07:24.647840 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dvj48\" is forbidden: User \"system:node:ip-10-0-135-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-67.ec2.internal' and this object" podUID="85301a7d-a72b-4b92-b266-176eb0c1625f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dvj48" Jul 10 19:07:24.650863 ip-10-0-135-67 kubenswrapper[2581]: E0710 19:07:24.650767 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jdjt9\" is forbidden: User \"system:node:ip-10-0-135-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-67.ec2.internal' and this object" podUID="e2cc3070-ce84-451b-9954-cea775baffdc" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jdjt9" Jul 10 19:07:24.666804 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:24.666496 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kr8fm"] Jul 10 19:07:24.669640 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:07:24.669604 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c6823d8_1cae_4102_89db_5c7d8e30d7aa.slice/crio-6dbb1e356e1cb232c504039c14e169c7e59397e6454e6c971eec824e960a1b6a WatchSource:0}: Error finding container 6dbb1e356e1cb232c504039c14e169c7e59397e6454e6c971eec824e960a1b6a: Status 404 returned error can't find the container with id 6dbb1e356e1cb232c504039c14e169c7e59397e6454e6c971eec824e960a1b6a Jul 10 19:07:24.699241 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:24.699209 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wqfnm"] Jul 10 19:07:24.701217 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:07:24.701192 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7981090_2b0a_4f1f_949b_218b720d5f7c.slice/crio-130ff94986a428eb896f06fdd195fd2b53e23e7a3a9ef4bc6d3d779ca63a4eb4 WatchSource:0}: Error finding container 130ff94986a428eb896f06fdd195fd2b53e23e7a3a9ef4bc6d3d779ca63a4eb4: Status 404 returned error can't find the container with id 130ff94986a428eb896f06fdd195fd2b53e23e7a3a9ef4bc6d3d779ca63a4eb4 Jul 10 19:07:24.723249 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:24.723224 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-j7rlm"] Jul 10 19:07:24.724190 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:07:24.724098 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18f05682_360f_40e3_a82a_b94fdb255a5a.slice/crio-56120bb4e90da0b05c63cb4ab6cb78d415be5333e4e8e0f3db66391cb5f2bb39 WatchSource:0}: Error finding container 56120bb4e90da0b05c63cb4ab6cb78d415be5333e4e8e0f3db66391cb5f2bb39: Status 404 returned error can't find the container with id 56120bb4e90da0b05c63cb4ab6cb78d415be5333e4e8e0f3db66391cb5f2bb39 Jul 10 19:07:24.745997 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:24.745969 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2x2cj\" (UniqueName: \"kubernetes.io/projected/85301a7d-a72b-4b92-b266-176eb0c1625f-kube-api-access-2x2cj\") pod \"85301a7d-a72b-4b92-b266-176eb0c1625f\" (UID: \"85301a7d-a72b-4b92-b266-176eb0c1625f\") " Jul 10 19:07:24.746107 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:24.746049 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6vg6\" (UniqueName: \"kubernetes.io/projected/e2cc3070-ce84-451b-9954-cea775baffdc-kube-api-access-j6vg6\") pod \"e2cc3070-ce84-451b-9954-cea775baffdc\" (UID: \"e2cc3070-ce84-451b-9954-cea775baffdc\") " Jul 10 19:07:24.746190 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:24.746163 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85301a7d-a72b-4b92-b266-176eb0c1625f-extensions-socket-volume\") pod \"85301a7d-a72b-4b92-b266-176eb0c1625f\" (UID: \"85301a7d-a72b-4b92-b266-176eb0c1625f\") " Jul 10 19:07:24.746902 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:24.746873 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85301a7d-a72b-4b92-b266-176eb0c1625f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "85301a7d-a72b-4b92-b266-176eb0c1625f" (UID: "85301a7d-a72b-4b92-b266-176eb0c1625f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:07:24.748517 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:24.748484 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85301a7d-a72b-4b92-b266-176eb0c1625f-kube-api-access-2x2cj" (OuterVolumeSpecName: "kube-api-access-2x2cj") pod "85301a7d-a72b-4b92-b266-176eb0c1625f" (UID: "85301a7d-a72b-4b92-b266-176eb0c1625f"). InnerVolumeSpecName "kube-api-access-2x2cj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:07:24.748594 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:24.748562 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2cc3070-ce84-451b-9954-cea775baffdc-kube-api-access-j6vg6" (OuterVolumeSpecName: "kube-api-access-j6vg6") pod "e2cc3070-ce84-451b-9954-cea775baffdc" (UID: "e2cc3070-ce84-451b-9954-cea775baffdc"). InnerVolumeSpecName "kube-api-access-j6vg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:07:24.847439 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:24.847399 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x2cj\" (UniqueName: \"kubernetes.io/projected/85301a7d-a72b-4b92-b266-176eb0c1625f-kube-api-access-2x2cj\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:07:24.847439 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:24.847440 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6vg6\" (UniqueName: \"kubernetes.io/projected/e2cc3070-ce84-451b-9954-cea775baffdc-kube-api-access-j6vg6\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:07:24.847621 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:24.847454 2581 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85301a7d-a72b-4b92-b266-176eb0c1625f-extensions-socket-volume\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:07:25.035803 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:25.035758 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kr8fm" event={"ID":"3c6823d8-1cae-4102-89db-5c7d8e30d7aa","Type":"ContainerStarted","Data":"127af6dead11c2d0fdd5bae9926d09986ff079b84721cabe51648db9e5946e8f"} Jul 10 19:07:25.036045 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:25.035824 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kr8fm" event={"ID":"3c6823d8-1cae-4102-89db-5c7d8e30d7aa","Type":"ContainerStarted","Data":"6dbb1e356e1cb232c504039c14e169c7e59397e6454e6c971eec824e960a1b6a"} Jul 10 19:07:25.036045 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:25.035856 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kr8fm" Jul 10 19:07:25.037510 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:25.037476 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qtvk4" event={"ID":"b260e5a5-56f3-47eb-9875-c2d0f740782a","Type":"ContainerStarted","Data":"5d6a15483194adf330096785c1da58a8a3911021375b6da96572fc78f3d3e152"} Jul 10 19:07:25.039518 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:25.039483 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-j7rlm" event={"ID":"18f05682-360f-40e3-a82a-b94fdb255a5a","Type":"ContainerStarted","Data":"80e17ea7a62a1225393471ebf9991d8143ec7bf5ed470eae821796171175c40a"} Jul 10 19:07:25.039642 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:25.039539 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-j7rlm" event={"ID":"18f05682-360f-40e3-a82a-b94fdb255a5a","Type":"ContainerStarted","Data":"56120bb4e90da0b05c63cb4ab6cb78d415be5333e4e8e0f3db66391cb5f2bb39"} Jul 10 19:07:25.039642 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:25.039584 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-j7rlm" Jul 10 19:07:25.041175 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:25.041149 2581 generic.go:356] "Generic (PLEG): container finished" podID="85301a7d-a72b-4b92-b266-176eb0c1625f" containerID="a0952420c607c5b63a69154f5bb9663646de747efb6e1b0c30736383ca1aad9d" exitCode=0 Jul 10 19:07:25.041287 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:25.041202 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dvj48" Jul 10 19:07:25.041351 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:25.041260 2581 scope.go:117] "RemoveContainer" containerID="a0952420c607c5b63a69154f5bb9663646de747efb6e1b0c30736383ca1aad9d" Jul 10 19:07:25.044196 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:25.044166 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wqfnm" event={"ID":"d7981090-2b0a-4f1f-949b-218b720d5f7c","Type":"ContainerStarted","Data":"d752d29f3c470d13c3b8c4c0566d4e71a30eec74afe2089bfb890be11ac60ede"} Jul 10 19:07:25.044327 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:25.044224 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wqfnm" event={"ID":"d7981090-2b0a-4f1f-949b-218b720d5f7c","Type":"ContainerStarted","Data":"130ff94986a428eb896f06fdd195fd2b53e23e7a3a9ef4bc6d3d779ca63a4eb4"} Jul 10 19:07:25.044396 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:25.044353 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wqfnm" Jul 10 19:07:25.045650 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:25.045625 2581 generic.go:356] "Generic (PLEG): container finished" podID="e2cc3070-ce84-451b-9954-cea775baffdc" containerID="532a012ad219b37f817d2c146b06f6b0b9d73f70f8f53399f5dd620637d3c44b" exitCode=0 Jul 10 19:07:25.045785 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:25.045657 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jdjt9" Jul 10 19:07:25.052500 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:25.052476 2581 scope.go:117] "RemoveContainer" containerID="a0952420c607c5b63a69154f5bb9663646de747efb6e1b0c30736383ca1aad9d" Jul 10 19:07:25.052806 ip-10-0-135-67 kubenswrapper[2581]: E0710 19:07:25.052785 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0952420c607c5b63a69154f5bb9663646de747efb6e1b0c30736383ca1aad9d\": container with ID starting with a0952420c607c5b63a69154f5bb9663646de747efb6e1b0c30736383ca1aad9d not found: ID does not exist" containerID="a0952420c607c5b63a69154f5bb9663646de747efb6e1b0c30736383ca1aad9d" Jul 10 19:07:25.052877 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:25.052815 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0952420c607c5b63a69154f5bb9663646de747efb6e1b0c30736383ca1aad9d"} err="failed to get container status \"a0952420c607c5b63a69154f5bb9663646de747efb6e1b0c30736383ca1aad9d\": rpc error: code = NotFound desc = could not find container \"a0952420c607c5b63a69154f5bb9663646de747efb6e1b0c30736383ca1aad9d\": container with ID starting with a0952420c607c5b63a69154f5bb9663646de747efb6e1b0c30736383ca1aad9d not found: ID does not exist" Jul 10 19:07:25.052877 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:25.052835 2581 scope.go:117] "RemoveContainer" containerID="532a012ad219b37f817d2c146b06f6b0b9d73f70f8f53399f5dd620637d3c44b" Jul 10 19:07:25.060712 ip-10-0-135-67 kubenswrapper[2581]: E0710 19:07:25.059720 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jdjt9\" is forbidden: User \"system:node:ip-10-0-135-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-67.ec2.internal' and this object" podUID="e2cc3070-ce84-451b-9954-cea775baffdc" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jdjt9" Jul 10 19:07:25.064196 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:25.062632 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kr8fm" podStartSLOduration=15.062612791 podStartE2EDuration="15.062612791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:07:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:07:25.05449784 +0000 UTC m=+529.098640341" watchObservedRunningTime="2026-07-10 19:07:25.062612791 +0000 UTC m=+529.106755293" Jul 10 19:07:25.064196 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:25.062887 2581 scope.go:117] "RemoveContainer" containerID="532a012ad219b37f817d2c146b06f6b0b9d73f70f8f53399f5dd620637d3c44b" Jul 10 19:07:25.064528 ip-10-0-135-67 kubenswrapper[2581]: E0710 19:07:25.064493 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"532a012ad219b37f817d2c146b06f6b0b9d73f70f8f53399f5dd620637d3c44b\": container with ID starting with 532a012ad219b37f817d2c146b06f6b0b9d73f70f8f53399f5dd620637d3c44b not found: ID does not exist" containerID="532a012ad219b37f817d2c146b06f6b0b9d73f70f8f53399f5dd620637d3c44b" Jul 10 19:07:25.064631 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:25.064548 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"532a012ad219b37f817d2c146b06f6b0b9d73f70f8f53399f5dd620637d3c44b"} err="failed to get container status \"532a012ad219b37f817d2c146b06f6b0b9d73f70f8f53399f5dd620637d3c44b\": rpc error: code = NotFound desc = could not find container \"532a012ad219b37f817d2c146b06f6b0b9d73f70f8f53399f5dd620637d3c44b\": container with ID starting with 532a012ad219b37f817d2c146b06f6b0b9d73f70f8f53399f5dd620637d3c44b not found: ID does not exist" Jul 10 19:07:25.068862 ip-10-0-135-67 kubenswrapper[2581]: E0710 19:07:25.068835 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dvj48\" is forbidden: User \"system:node:ip-10-0-135-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-67.ec2.internal' and this object" podUID="85301a7d-a72b-4b92-b266-176eb0c1625f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dvj48" Jul 10 19:07:25.070944 ip-10-0-135-67 kubenswrapper[2581]: E0710 19:07:25.070920 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jdjt9\" is forbidden: User \"system:node:ip-10-0-135-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-67.ec2.internal' and this object" podUID="e2cc3070-ce84-451b-9954-cea775baffdc" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jdjt9" Jul 10 19:07:25.073175 ip-10-0-135-67 kubenswrapper[2581]: E0710 19:07:25.073155 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dvj48\" is forbidden: User \"system:node:ip-10-0-135-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-67.ec2.internal' and this object" podUID="85301a7d-a72b-4b92-b266-176eb0c1625f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dvj48" Jul 10 19:07:25.091486 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:25.091440 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qtvk4" podStartSLOduration=1.860282268 podStartE2EDuration="25.091425933s" totalImagesPullingTime="23.231143665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:07:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:07:25.089695588 +0000 UTC m=+529.133838097" watchObservedRunningTime="2026-07-10 19:07:25.091425933 +0000 UTC m=+529.135568433" Jul 10 19:07:25.108298 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:25.108248 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wqfnm" podStartSLOduration=13.108231327 podStartE2EDuration="13.108231327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:07:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:07:25.107895088 +0000 UTC m=+529.152037590" watchObservedRunningTime="2026-07-10 19:07:25.108231327 +0000 UTC m=+529.152373810" Jul 10 19:07:25.133490 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:25.133389 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-j7rlm" podStartSLOduration=15.133367361 podStartE2EDuration="15.133367361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:07:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:07:25.128553365 +0000 UTC m=+529.172695866" watchObservedRunningTime="2026-07-10 19:07:25.133367361 +0000 UTC m=+529.177509863" Jul 10 19:07:26.541056 ip-10-0-135-67 kubenswrapper[2581]: E0710 19:07:26.541008 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dvj48\" is forbidden: User \"system:node:ip-10-0-135-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-67.ec2.internal' and this object" podUID="85301a7d-a72b-4b92-b266-176eb0c1625f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dvj48" Jul 10 19:07:26.542821 ip-10-0-135-67 kubenswrapper[2581]: E0710 19:07:26.542788 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jdjt9\" is forbidden: User \"system:node:ip-10-0-135-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-67.ec2.internal' and this object" podUID="e2cc3070-ce84-451b-9954-cea775baffdc" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jdjt9" Jul 10 19:07:26.543054 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:26.543034 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85301a7d-a72b-4b92-b266-176eb0c1625f" path="/var/lib/kubelet/pods/85301a7d-a72b-4b92-b266-176eb0c1625f/volumes" Jul 10 19:07:26.543367 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:26.543354 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2cc3070-ce84-451b-9954-cea775baffdc" path="/var/lib/kubelet/pods/e2cc3070-ce84-451b-9954-cea775baffdc/volumes" Jul 10 19:07:36.056892 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:36.056803 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kr8fm" Jul 10 19:07:36.057323 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:36.056962 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-j7rlm" Jul 10 19:07:36.057323 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:36.056998 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wqfnm" Jul 10 19:07:36.145584 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:36.145548 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-j7rlm"] Jul 10 19:07:36.145833 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:36.145805 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-j7rlm" podUID="18f05682-360f-40e3-a82a-b94fdb255a5a" containerName="manager" containerID="cri-o://80e17ea7a62a1225393471ebf9991d8143ec7bf5ed470eae821796171175c40a" gracePeriod=10 Jul 10 19:07:36.402779 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:36.402262 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-j7rlm" Jul 10 19:07:36.442411 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:36.442384 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmlr4\" (UniqueName: \"kubernetes.io/projected/18f05682-360f-40e3-a82a-b94fdb255a5a-kube-api-access-tmlr4\") pod \"18f05682-360f-40e3-a82a-b94fdb255a5a\" (UID: \"18f05682-360f-40e3-a82a-b94fdb255a5a\") " Jul 10 19:07:36.442557 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:36.442456 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/18f05682-360f-40e3-a82a-b94fdb255a5a-extensions-socket-volume\") pod \"18f05682-360f-40e3-a82a-b94fdb255a5a\" (UID: \"18f05682-360f-40e3-a82a-b94fdb255a5a\") " Jul 10 19:07:36.444774 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:36.442833 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18f05682-360f-40e3-a82a-b94fdb255a5a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "18f05682-360f-40e3-a82a-b94fdb255a5a" (UID: "18f05682-360f-40e3-a82a-b94fdb255a5a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:07:36.446529 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:36.446497 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18f05682-360f-40e3-a82a-b94fdb255a5a-kube-api-access-tmlr4" (OuterVolumeSpecName: "kube-api-access-tmlr4") pod "18f05682-360f-40e3-a82a-b94fdb255a5a" (UID: "18f05682-360f-40e3-a82a-b94fdb255a5a"). InnerVolumeSpecName "kube-api-access-tmlr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:07:36.542954 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:36.542923 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmlr4\" (UniqueName: \"kubernetes.io/projected/18f05682-360f-40e3-a82a-b94fdb255a5a-kube-api-access-tmlr4\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:07:36.542954 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:36.542950 2581 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/18f05682-360f-40e3-a82a-b94fdb255a5a-extensions-socket-volume\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:07:37.112002 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:37.111965 2581 generic.go:356] "Generic (PLEG): container finished" podID="18f05682-360f-40e3-a82a-b94fdb255a5a" containerID="80e17ea7a62a1225393471ebf9991d8143ec7bf5ed470eae821796171175c40a" exitCode=0 Jul 10 19:07:37.112438 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:37.112020 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-j7rlm" Jul 10 19:07:37.112438 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:37.112023 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-j7rlm" event={"ID":"18f05682-360f-40e3-a82a-b94fdb255a5a","Type":"ContainerDied","Data":"80e17ea7a62a1225393471ebf9991d8143ec7bf5ed470eae821796171175c40a"} Jul 10 19:07:37.112438 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:37.112145 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-j7rlm" event={"ID":"18f05682-360f-40e3-a82a-b94fdb255a5a","Type":"ContainerDied","Data":"56120bb4e90da0b05c63cb4ab6cb78d415be5333e4e8e0f3db66391cb5f2bb39"} Jul 10 19:07:37.112438 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:37.112176 2581 scope.go:117] "RemoveContainer" containerID="80e17ea7a62a1225393471ebf9991d8143ec7bf5ed470eae821796171175c40a" Jul 10 19:07:37.121553 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:37.121533 2581 scope.go:117] "RemoveContainer" containerID="80e17ea7a62a1225393471ebf9991d8143ec7bf5ed470eae821796171175c40a" Jul 10 19:07:37.121836 ip-10-0-135-67 kubenswrapper[2581]: E0710 19:07:37.121814 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80e17ea7a62a1225393471ebf9991d8143ec7bf5ed470eae821796171175c40a\": container with ID starting with 80e17ea7a62a1225393471ebf9991d8143ec7bf5ed470eae821796171175c40a not found: ID does not exist" containerID="80e17ea7a62a1225393471ebf9991d8143ec7bf5ed470eae821796171175c40a" Jul 10 19:07:37.121899 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:37.121850 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80e17ea7a62a1225393471ebf9991d8143ec7bf5ed470eae821796171175c40a"} err="failed to get container status \"80e17ea7a62a1225393471ebf9991d8143ec7bf5ed470eae821796171175c40a\": rpc error: code = NotFound desc = could not find container \"80e17ea7a62a1225393471ebf9991d8143ec7bf5ed470eae821796171175c40a\": container with ID starting with 80e17ea7a62a1225393471ebf9991d8143ec7bf5ed470eae821796171175c40a not found: ID does not exist" Jul 10 19:07:37.131418 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:37.131380 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-j7rlm"] Jul 10 19:07:37.138357 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:37.138333 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-j7rlm"] Jul 10 19:07:38.543206 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:38.543173 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18f05682-360f-40e3-a82a-b94fdb255a5a" path="/var/lib/kubelet/pods/18f05682-360f-40e3-a82a-b94fdb255a5a/volumes" Jul 10 19:07:54.868428 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:54.868389 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-zt22t"] Jul 10 19:07:54.868873 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:54.868804 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18f05682-360f-40e3-a82a-b94fdb255a5a" containerName="manager" Jul 10 19:07:54.868873 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:54.868817 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="18f05682-360f-40e3-a82a-b94fdb255a5a" containerName="manager" Jul 10 19:07:54.868963 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:54.868889 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="18f05682-360f-40e3-a82a-b94fdb255a5a" containerName="manager" Jul 10 19:07:54.909262 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:54.909228 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-zt22t"] Jul 10 19:07:54.909434 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:54.909357 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-zt22t" Jul 10 19:07:54.911843 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:54.911815 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 10 19:07:54.963206 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:54.963170 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-zt22t"] Jul 10 19:07:54.999072 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:54.999037 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f2cc5f7e-2bb9-4915-a2ba-dbb6a18a10e3-config-file\") pod \"limitador-limitador-64c94fd64f-zt22t\" (UID: \"f2cc5f7e-2bb9-4915-a2ba-dbb6a18a10e3\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-zt22t" Jul 10 19:07:54.999546 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:54.999101 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvx9z\" (UniqueName: \"kubernetes.io/projected/f2cc5f7e-2bb9-4915-a2ba-dbb6a18a10e3-kube-api-access-dvx9z\") pod \"limitador-limitador-64c94fd64f-zt22t\" (UID: \"f2cc5f7e-2bb9-4915-a2ba-dbb6a18a10e3\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-zt22t" Jul 10 19:07:55.100453 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:55.100411 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f2cc5f7e-2bb9-4915-a2ba-dbb6a18a10e3-config-file\") pod \"limitador-limitador-64c94fd64f-zt22t\" (UID: \"f2cc5f7e-2bb9-4915-a2ba-dbb6a18a10e3\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-zt22t" Jul 10 19:07:55.100625 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:55.100492 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvx9z\" (UniqueName: \"kubernetes.io/projected/f2cc5f7e-2bb9-4915-a2ba-dbb6a18a10e3-kube-api-access-dvx9z\") pod \"limitador-limitador-64c94fd64f-zt22t\" (UID: \"f2cc5f7e-2bb9-4915-a2ba-dbb6a18a10e3\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-zt22t" Jul 10 19:07:55.101056 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:55.101038 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f2cc5f7e-2bb9-4915-a2ba-dbb6a18a10e3-config-file\") pod \"limitador-limitador-64c94fd64f-zt22t\" (UID: \"f2cc5f7e-2bb9-4915-a2ba-dbb6a18a10e3\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-zt22t" Jul 10 19:07:55.112793 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:55.112768 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvx9z\" (UniqueName: \"kubernetes.io/projected/f2cc5f7e-2bb9-4915-a2ba-dbb6a18a10e3-kube-api-access-dvx9z\") pod \"limitador-limitador-64c94fd64f-zt22t\" (UID: \"f2cc5f7e-2bb9-4915-a2ba-dbb6a18a10e3\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-zt22t" Jul 10 19:07:55.220277 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:55.220197 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-zt22t" Jul 10 19:07:55.364427 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:55.364402 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-zt22t"] Jul 10 19:07:55.366055 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:07:55.366027 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2cc5f7e_2bb9_4915_a2ba_dbb6a18a10e3.slice/crio-6462adc2650e2700caee180b933238f0a61d162fb9819c6ef94c246b66dd8c23 WatchSource:0}: Error finding container 6462adc2650e2700caee180b933238f0a61d162fb9819c6ef94c246b66dd8c23: Status 404 returned error can't find the container with id 6462adc2650e2700caee180b933238f0a61d162fb9819c6ef94c246b66dd8c23 Jul 10 19:07:55.782099 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:55.782059 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-nwjh5"] Jul 10 19:07:55.786824 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:55.786800 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-nwjh5" Jul 10 19:07:55.789275 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:55.789252 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-8d4jn\"" Jul 10 19:07:55.790659 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:55.790627 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-nwjh5"] Jul 10 19:07:55.909237 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:55.909206 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2pgf\" (UniqueName: \"kubernetes.io/projected/1b245ae2-c7d9-416f-98df-ce3944b61b51-kube-api-access-k2pgf\") pod \"authorino-75d556bd77-nwjh5\" (UID: \"1b245ae2-c7d9-416f-98df-ce3944b61b51\") " pod="kuadrant-system/authorino-75d556bd77-nwjh5" Jul 10 19:07:56.009755 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:56.009710 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2pgf\" (UniqueName: \"kubernetes.io/projected/1b245ae2-c7d9-416f-98df-ce3944b61b51-kube-api-access-k2pgf\") pod \"authorino-75d556bd77-nwjh5\" (UID: \"1b245ae2-c7d9-416f-98df-ce3944b61b51\") " pod="kuadrant-system/authorino-75d556bd77-nwjh5" Jul 10 19:07:56.017972 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:56.017947 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2pgf\" (UniqueName: \"kubernetes.io/projected/1b245ae2-c7d9-416f-98df-ce3944b61b51-kube-api-access-k2pgf\") pod \"authorino-75d556bd77-nwjh5\" (UID: \"1b245ae2-c7d9-416f-98df-ce3944b61b51\") " pod="kuadrant-system/authorino-75d556bd77-nwjh5" Jul 10 19:07:56.098374 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:56.098298 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-nwjh5" Jul 10 19:07:56.227407 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:56.227288 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-zt22t" event={"ID":"f2cc5f7e-2bb9-4915-a2ba-dbb6a18a10e3","Type":"ContainerStarted","Data":"6462adc2650e2700caee180b933238f0a61d162fb9819c6ef94c246b66dd8c23"} Jul 10 19:07:56.267675 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:56.267600 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-nwjh5"] Jul 10 19:07:56.273161 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:07:56.273124 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b245ae2_c7d9_416f_98df_ce3944b61b51.slice/crio-a9e78157bce560591bbd6ae8045dd20e35ae329d658b32e0690f8a3353f3bdfe WatchSource:0}: Error finding container a9e78157bce560591bbd6ae8045dd20e35ae329d658b32e0690f8a3353f3bdfe: Status 404 returned error can't find the container with id a9e78157bce560591bbd6ae8045dd20e35ae329d658b32e0690f8a3353f3bdfe Jul 10 19:07:57.239700 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:07:57.239634 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-nwjh5" event={"ID":"1b245ae2-c7d9-416f-98df-ce3944b61b51","Type":"ContainerStarted","Data":"a9e78157bce560591bbd6ae8045dd20e35ae329d658b32e0690f8a3353f3bdfe"} Jul 10 19:08:00.259725 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:00.259664 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-nwjh5" event={"ID":"1b245ae2-c7d9-416f-98df-ce3944b61b51","Type":"ContainerStarted","Data":"1a48e6ce5d2a3aeabd29e2b42ab98e1baa9fa105d46c550ba32db4cb5e853a41"} Jul 10 19:08:00.261172 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:00.261141 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-zt22t" event={"ID":"f2cc5f7e-2bb9-4915-a2ba-dbb6a18a10e3","Type":"ContainerStarted","Data":"a93d6d96ec57442092cb7847080732d029c638f926dcc0969a49e44c20ffd795"} Jul 10 19:08:00.261300 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:00.261236 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-zt22t" Jul 10 19:08:00.275290 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:00.275246 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-nwjh5" podStartSLOduration=1.49186792 podStartE2EDuration="5.27523244s" totalImagesPullingTime="3.78336452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:07:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:08:00.274281342 +0000 UTC m=+564.318423853" watchObservedRunningTime="2026-07-10 19:08:00.27523244 +0000 UTC m=+564.319374979" Jul 10 19:08:00.291858 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:00.291797 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-zt22t" podStartSLOduration=1.6647699409999999 podStartE2EDuration="6.291780054s" totalImagesPullingTime="4.627010113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:07:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:08:00.290699874 +0000 UTC m=+564.334842404" watchObservedRunningTime="2026-07-10 19:08:00.291780054 +0000 UTC m=+564.335922555" Jul 10 19:08:11.268154 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:11.268124 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-zt22t" Jul 10 19:08:11.724734 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:11.724616 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-zt22t"] Jul 10 19:08:11.724934 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:11.724893 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-zt22t" podUID="f2cc5f7e-2bb9-4915-a2ba-dbb6a18a10e3" containerName="limitador" containerID="cri-o://a93d6d96ec57442092cb7847080732d029c638f926dcc0969a49e44c20ffd795" gracePeriod=30 Jul 10 19:08:12.277559 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:12.277536 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-zt22t" Jul 10 19:08:12.324433 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:12.324332 2581 generic.go:356] "Generic (PLEG): container finished" podID="f2cc5f7e-2bb9-4915-a2ba-dbb6a18a10e3" containerID="a93d6d96ec57442092cb7847080732d029c638f926dcc0969a49e44c20ffd795" exitCode=0 Jul 10 19:08:12.324433 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:12.324394 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-zt22t" Jul 10 19:08:12.324433 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:12.324410 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-zt22t" event={"ID":"f2cc5f7e-2bb9-4915-a2ba-dbb6a18a10e3","Type":"ContainerDied","Data":"a93d6d96ec57442092cb7847080732d029c638f926dcc0969a49e44c20ffd795"} Jul 10 19:08:12.324704 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:12.324452 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-zt22t" event={"ID":"f2cc5f7e-2bb9-4915-a2ba-dbb6a18a10e3","Type":"ContainerDied","Data":"6462adc2650e2700caee180b933238f0a61d162fb9819c6ef94c246b66dd8c23"} Jul 10 19:08:12.324704 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:12.324477 2581 scope.go:117] "RemoveContainer" containerID="a93d6d96ec57442092cb7847080732d029c638f926dcc0969a49e44c20ffd795" Jul 10 19:08:12.333283 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:12.333260 2581 scope.go:117] "RemoveContainer" containerID="a93d6d96ec57442092cb7847080732d029c638f926dcc0969a49e44c20ffd795" Jul 10 19:08:12.333571 ip-10-0-135-67 kubenswrapper[2581]: E0710 19:08:12.333545 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a93d6d96ec57442092cb7847080732d029c638f926dcc0969a49e44c20ffd795\": container with ID starting with a93d6d96ec57442092cb7847080732d029c638f926dcc0969a49e44c20ffd795 not found: ID does not exist" containerID="a93d6d96ec57442092cb7847080732d029c638f926dcc0969a49e44c20ffd795" Jul 10 19:08:12.333642 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:12.333582 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a93d6d96ec57442092cb7847080732d029c638f926dcc0969a49e44c20ffd795"} err="failed to get container status \"a93d6d96ec57442092cb7847080732d029c638f926dcc0969a49e44c20ffd795\": rpc error: code = NotFound desc = could not find container \"a93d6d96ec57442092cb7847080732d029c638f926dcc0969a49e44c20ffd795\": container with ID starting with a93d6d96ec57442092cb7847080732d029c638f926dcc0969a49e44c20ffd795 not found: ID does not exist" Jul 10 19:08:12.356927 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:12.356893 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f2cc5f7e-2bb9-4915-a2ba-dbb6a18a10e3-config-file\") pod \"f2cc5f7e-2bb9-4915-a2ba-dbb6a18a10e3\" (UID: \"f2cc5f7e-2bb9-4915-a2ba-dbb6a18a10e3\") " Jul 10 19:08:12.357036 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:12.356981 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvx9z\" (UniqueName: \"kubernetes.io/projected/f2cc5f7e-2bb9-4915-a2ba-dbb6a18a10e3-kube-api-access-dvx9z\") pod \"f2cc5f7e-2bb9-4915-a2ba-dbb6a18a10e3\" (UID: \"f2cc5f7e-2bb9-4915-a2ba-dbb6a18a10e3\") " Jul 10 19:08:12.357289 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:12.357260 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2cc5f7e-2bb9-4915-a2ba-dbb6a18a10e3-config-file" (OuterVolumeSpecName: "config-file") pod "f2cc5f7e-2bb9-4915-a2ba-dbb6a18a10e3" (UID: "f2cc5f7e-2bb9-4915-a2ba-dbb6a18a10e3"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:08:12.359009 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:12.358987 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2cc5f7e-2bb9-4915-a2ba-dbb6a18a10e3-kube-api-access-dvx9z" (OuterVolumeSpecName: "kube-api-access-dvx9z") pod "f2cc5f7e-2bb9-4915-a2ba-dbb6a18a10e3" (UID: "f2cc5f7e-2bb9-4915-a2ba-dbb6a18a10e3"). InnerVolumeSpecName "kube-api-access-dvx9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:08:12.458093 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:12.458053 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvx9z\" (UniqueName: \"kubernetes.io/projected/f2cc5f7e-2bb9-4915-a2ba-dbb6a18a10e3-kube-api-access-dvx9z\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:08:12.458093 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:12.458085 2581 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f2cc5f7e-2bb9-4915-a2ba-dbb6a18a10e3-config-file\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:08:12.640332 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:12.640250 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-zt22t"] Jul 10 19:08:12.643854 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:12.643819 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-zt22t"] Jul 10 19:08:14.542094 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:14.542057 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2cc5f7e-2bb9-4915-a2ba-dbb6a18a10e3" path="/var/lib/kubelet/pods/f2cc5f7e-2bb9-4915-a2ba-dbb6a18a10e3/volumes" Jul 10 19:08:16.452147 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:16.452112 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-rzrns"] Jul 10 19:08:16.452529 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:16.452469 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2cc5f7e-2bb9-4915-a2ba-dbb6a18a10e3" containerName="limitador" Jul 10 19:08:16.452529 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:16.452480 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2cc5f7e-2bb9-4915-a2ba-dbb6a18a10e3" containerName="limitador" Jul 10 19:08:16.452609 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:16.452552 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2cc5f7e-2bb9-4915-a2ba-dbb6a18a10e3" containerName="limitador" Jul 10 19:08:16.455272 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:16.455252 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rzrns" Jul 10 19:08:16.457809 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:16.457787 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 10 19:08:16.458828 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:16.458808 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-dt9q2\"" Jul 10 19:08:16.458941 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:16.458853 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 10 19:08:16.458941 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:16.458876 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 19:08:16.465803 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:16.465778 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-rzrns"] Jul 10 19:08:16.593125 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:16.593077 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3b5943f0-10c9-4290-8710-a28dd4fd3aab-data\") pod \"postgres-69fb4594cf-rzrns\" (UID: \"3b5943f0-10c9-4290-8710-a28dd4fd3aab\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rzrns" Jul 10 19:08:16.593125 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:16.593123 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px74r\" (UniqueName: \"kubernetes.io/projected/3b5943f0-10c9-4290-8710-a28dd4fd3aab-kube-api-access-px74r\") pod \"postgres-69fb4594cf-rzrns\" (UID: \"3b5943f0-10c9-4290-8710-a28dd4fd3aab\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rzrns" Jul 10 19:08:16.694407 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:16.694362 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3b5943f0-10c9-4290-8710-a28dd4fd3aab-data\") pod \"postgres-69fb4594cf-rzrns\" (UID: \"3b5943f0-10c9-4290-8710-a28dd4fd3aab\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rzrns" Jul 10 19:08:16.694407 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:16.694407 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px74r\" (UniqueName: \"kubernetes.io/projected/3b5943f0-10c9-4290-8710-a28dd4fd3aab-kube-api-access-px74r\") pod \"postgres-69fb4594cf-rzrns\" (UID: \"3b5943f0-10c9-4290-8710-a28dd4fd3aab\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rzrns" Jul 10 19:08:16.694811 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:16.694791 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3b5943f0-10c9-4290-8710-a28dd4fd3aab-data\") pod \"postgres-69fb4594cf-rzrns\" (UID: \"3b5943f0-10c9-4290-8710-a28dd4fd3aab\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rzrns" Jul 10 19:08:16.704019 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:16.703952 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-px74r\" (UniqueName: \"kubernetes.io/projected/3b5943f0-10c9-4290-8710-a28dd4fd3aab-kube-api-access-px74r\") pod \"postgres-69fb4594cf-rzrns\" (UID: \"3b5943f0-10c9-4290-8710-a28dd4fd3aab\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rzrns" Jul 10 19:08:16.767546 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:16.767512 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rzrns" Jul 10 19:08:16.902023 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:16.901999 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-rzrns"] Jul 10 19:08:16.904155 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:08:16.904123 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b5943f0_10c9_4290_8710_a28dd4fd3aab.slice/crio-f24eb125c0bec7e0de4c06781cabe6f109f707864e61af1bf3fe42b66a44c1ea WatchSource:0}: Error finding container f24eb125c0bec7e0de4c06781cabe6f109f707864e61af1bf3fe42b66a44c1ea: Status 404 returned error can't find the container with id f24eb125c0bec7e0de4c06781cabe6f109f707864e61af1bf3fe42b66a44c1ea Jul 10 19:08:17.353799 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:17.353757 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rzrns" event={"ID":"3b5943f0-10c9-4290-8710-a28dd4fd3aab","Type":"ContainerStarted","Data":"f24eb125c0bec7e0de4c06781cabe6f109f707864e61af1bf3fe42b66a44c1ea"} Jul 10 19:08:22.386970 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:22.386933 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rzrns" event={"ID":"3b5943f0-10c9-4290-8710-a28dd4fd3aab","Type":"ContainerStarted","Data":"b713069f8551437fedc2c67015c45fd147bc3eb02a134b03ea139b3f2aad14f3"} Jul 10 19:08:22.387333 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:22.387052 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rzrns" Jul 10 19:08:22.405640 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:22.405590 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rzrns" podStartSLOduration=1.575783997 podStartE2EDuration="6.405573588s" totalImagesPullingTime="4.829789591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:08:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:08:22.403879519 +0000 UTC m=+586.448022019" watchObservedRunningTime="2026-07-10 19:08:22.405573588 +0000 UTC m=+586.449716088" Jul 10 19:08:28.422290 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:28.422261 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rzrns" Jul 10 19:08:30.835994 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:30.835961 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-nwjh5"] Jul 10 19:08:30.836408 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:30.836164 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-nwjh5" podUID="1b245ae2-c7d9-416f-98df-ce3944b61b51" containerName="authorino" containerID="cri-o://1a48e6ce5d2a3aeabd29e2b42ab98e1baa9fa105d46c550ba32db4cb5e853a41" gracePeriod=30 Jul 10 19:08:31.133038 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:31.133007 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-nwjh5" Jul 10 19:08:31.204930 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:31.204897 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2pgf\" (UniqueName: \"kubernetes.io/projected/1b245ae2-c7d9-416f-98df-ce3944b61b51-kube-api-access-k2pgf\") pod \"1b245ae2-c7d9-416f-98df-ce3944b61b51\" (UID: \"1b245ae2-c7d9-416f-98df-ce3944b61b51\") " Jul 10 19:08:31.206914 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:31.206878 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b245ae2-c7d9-416f-98df-ce3944b61b51-kube-api-access-k2pgf" (OuterVolumeSpecName: "kube-api-access-k2pgf") pod "1b245ae2-c7d9-416f-98df-ce3944b61b51" (UID: "1b245ae2-c7d9-416f-98df-ce3944b61b51"). InnerVolumeSpecName "kube-api-access-k2pgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:08:31.306152 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:31.306112 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2pgf\" (UniqueName: \"kubernetes.io/projected/1b245ae2-c7d9-416f-98df-ce3944b61b51-kube-api-access-k2pgf\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:08:31.437040 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:31.436953 2581 generic.go:356] "Generic (PLEG): container finished" podID="1b245ae2-c7d9-416f-98df-ce3944b61b51" containerID="1a48e6ce5d2a3aeabd29e2b42ab98e1baa9fa105d46c550ba32db4cb5e853a41" exitCode=0 Jul 10 19:08:31.437040 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:31.436996 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-nwjh5" Jul 10 19:08:31.437249 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:31.437000 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-nwjh5" event={"ID":"1b245ae2-c7d9-416f-98df-ce3944b61b51","Type":"ContainerDied","Data":"1a48e6ce5d2a3aeabd29e2b42ab98e1baa9fa105d46c550ba32db4cb5e853a41"} Jul 10 19:08:31.437249 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:31.437118 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-nwjh5" event={"ID":"1b245ae2-c7d9-416f-98df-ce3944b61b51","Type":"ContainerDied","Data":"a9e78157bce560591bbd6ae8045dd20e35ae329d658b32e0690f8a3353f3bdfe"} Jul 10 19:08:31.437249 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:31.437151 2581 scope.go:117] "RemoveContainer" containerID="1a48e6ce5d2a3aeabd29e2b42ab98e1baa9fa105d46c550ba32db4cb5e853a41" Jul 10 19:08:31.446618 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:31.446600 2581 scope.go:117] "RemoveContainer" containerID="1a48e6ce5d2a3aeabd29e2b42ab98e1baa9fa105d46c550ba32db4cb5e853a41" Jul 10 19:08:31.446987 ip-10-0-135-67 kubenswrapper[2581]: E0710 19:08:31.446968 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a48e6ce5d2a3aeabd29e2b42ab98e1baa9fa105d46c550ba32db4cb5e853a41\": container with ID starting with 1a48e6ce5d2a3aeabd29e2b42ab98e1baa9fa105d46c550ba32db4cb5e853a41 not found: ID does not exist" containerID="1a48e6ce5d2a3aeabd29e2b42ab98e1baa9fa105d46c550ba32db4cb5e853a41" Jul 10 19:08:31.447064 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:31.446998 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a48e6ce5d2a3aeabd29e2b42ab98e1baa9fa105d46c550ba32db4cb5e853a41"} err="failed to get container status \"1a48e6ce5d2a3aeabd29e2b42ab98e1baa9fa105d46c550ba32db4cb5e853a41\": rpc error: code = NotFound desc = could not find container \"1a48e6ce5d2a3aeabd29e2b42ab98e1baa9fa105d46c550ba32db4cb5e853a41\": container with ID starting with 1a48e6ce5d2a3aeabd29e2b42ab98e1baa9fa105d46c550ba32db4cb5e853a41 not found: ID does not exist" Jul 10 19:08:31.462029 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:31.462003 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-nwjh5"] Jul 10 19:08:31.469505 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:31.469482 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-nwjh5"] Jul 10 19:08:32.541883 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:32.541847 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b245ae2-c7d9-416f-98df-ce3944b61b51" path="/var/lib/kubelet/pods/1b245ae2-c7d9-416f-98df-ce3944b61b51/volumes" Jul 10 19:08:34.981579 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:34.981539 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-84f7ddd9bc-5xsvn"] Jul 10 19:08:34.982155 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:34.981919 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b245ae2-c7d9-416f-98df-ce3944b61b51" containerName="authorino" Jul 10 19:08:34.982155 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:34.981935 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b245ae2-c7d9-416f-98df-ce3944b61b51" containerName="authorino" Jul 10 19:08:34.982155 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:34.982060 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b245ae2-c7d9-416f-98df-ce3944b61b51" containerName="authorino" Jul 10 19:08:35.163915 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:35.163881 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84f7ddd9bc-5xsvn"] Jul 10 19:08:35.164083 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:35.163927 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-77d7f68d48-227th"] Jul 10 19:08:35.164083 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:35.164014 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84f7ddd9bc-5xsvn" Jul 10 19:08:35.166592 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:35.166562 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-4b97p\"" Jul 10 19:08:35.166771 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:35.166611 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 10 19:08:35.166880 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:35.166853 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 10 19:08:35.183278 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:35.183246 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77d7f68d48-227th"] Jul 10 19:08:35.183416 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:35.183362 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77d7f68d48-227th" Jul 10 19:08:35.241712 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:35.241594 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xrlz\" (UniqueName: \"kubernetes.io/projected/35277ea5-79dc-4107-b8b3-ec94330189ef-kube-api-access-6xrlz\") pod \"maas-controller-84f7ddd9bc-5xsvn\" (UID: \"35277ea5-79dc-4107-b8b3-ec94330189ef\") " pod="opendatahub/maas-controller-84f7ddd9bc-5xsvn" Jul 10 19:08:35.241712 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:35.241631 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/79af4c9e-1861-49bc-9d77-e07174ba3495-webhook-cert\") pod \"maas-controller-77d7f68d48-227th\" (UID: \"79af4c9e-1861-49bc-9d77-e07174ba3495\") " pod="opendatahub/maas-controller-77d7f68d48-227th" Jul 10 19:08:35.241901 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:35.241744 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpn6m\" (UniqueName: \"kubernetes.io/projected/79af4c9e-1861-49bc-9d77-e07174ba3495-kube-api-access-vpn6m\") pod \"maas-controller-77d7f68d48-227th\" (UID: \"79af4c9e-1861-49bc-9d77-e07174ba3495\") " pod="opendatahub/maas-controller-77d7f68d48-227th" Jul 10 19:08:35.241901 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:35.241776 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/35277ea5-79dc-4107-b8b3-ec94330189ef-webhook-cert\") pod \"maas-controller-84f7ddd9bc-5xsvn\" (UID: \"35277ea5-79dc-4107-b8b3-ec94330189ef\") " pod="opendatahub/maas-controller-84f7ddd9bc-5xsvn" Jul 10 19:08:35.342520 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:35.342484 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpn6m\" (UniqueName: \"kubernetes.io/projected/79af4c9e-1861-49bc-9d77-e07174ba3495-kube-api-access-vpn6m\") pod \"maas-controller-77d7f68d48-227th\" (UID: \"79af4c9e-1861-49bc-9d77-e07174ba3495\") " pod="opendatahub/maas-controller-77d7f68d48-227th" Jul 10 19:08:35.342697 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:35.342524 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/35277ea5-79dc-4107-b8b3-ec94330189ef-webhook-cert\") pod \"maas-controller-84f7ddd9bc-5xsvn\" (UID: \"35277ea5-79dc-4107-b8b3-ec94330189ef\") " pod="opendatahub/maas-controller-84f7ddd9bc-5xsvn" Jul 10 19:08:35.342697 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:35.342607 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xrlz\" (UniqueName: \"kubernetes.io/projected/35277ea5-79dc-4107-b8b3-ec94330189ef-kube-api-access-6xrlz\") pod \"maas-controller-84f7ddd9bc-5xsvn\" (UID: \"35277ea5-79dc-4107-b8b3-ec94330189ef\") " pod="opendatahub/maas-controller-84f7ddd9bc-5xsvn" Jul 10 19:08:35.342697 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:35.342633 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/79af4c9e-1861-49bc-9d77-e07174ba3495-webhook-cert\") pod \"maas-controller-77d7f68d48-227th\" (UID: \"79af4c9e-1861-49bc-9d77-e07174ba3495\") " pod="opendatahub/maas-controller-77d7f68d48-227th" Jul 10 19:08:35.345116 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:35.345082 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/35277ea5-79dc-4107-b8b3-ec94330189ef-webhook-cert\") pod \"maas-controller-84f7ddd9bc-5xsvn\" (UID: \"35277ea5-79dc-4107-b8b3-ec94330189ef\") " pod="opendatahub/maas-controller-84f7ddd9bc-5xsvn" Jul 10 19:08:35.345211 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:35.345137 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/79af4c9e-1861-49bc-9d77-e07174ba3495-webhook-cert\") pod \"maas-controller-77d7f68d48-227th\" (UID: \"79af4c9e-1861-49bc-9d77-e07174ba3495\") " pod="opendatahub/maas-controller-77d7f68d48-227th" Jul 10 19:08:35.350891 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:35.350869 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpn6m\" (UniqueName: \"kubernetes.io/projected/79af4c9e-1861-49bc-9d77-e07174ba3495-kube-api-access-vpn6m\") pod \"maas-controller-77d7f68d48-227th\" (UID: \"79af4c9e-1861-49bc-9d77-e07174ba3495\") " pod="opendatahub/maas-controller-77d7f68d48-227th" Jul 10 19:08:35.351050 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:35.351029 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xrlz\" (UniqueName: \"kubernetes.io/projected/35277ea5-79dc-4107-b8b3-ec94330189ef-kube-api-access-6xrlz\") pod \"maas-controller-84f7ddd9bc-5xsvn\" (UID: \"35277ea5-79dc-4107-b8b3-ec94330189ef\") " pod="opendatahub/maas-controller-84f7ddd9bc-5xsvn" Jul 10 19:08:35.476114 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:35.476082 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84f7ddd9bc-5xsvn" Jul 10 19:08:35.492222 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:35.492162 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77d7f68d48-227th" Jul 10 19:08:35.641677 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:35.641651 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84f7ddd9bc-5xsvn"] Jul 10 19:08:35.645773 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:08:35.645721 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35277ea5_79dc_4107_b8b3_ec94330189ef.slice/crio-80f1bdd5a48e9c889a83a71399b182ddb67c1c6d675b97597809b8ba1ddc7d16 WatchSource:0}: Error finding container 80f1bdd5a48e9c889a83a71399b182ddb67c1c6d675b97597809b8ba1ddc7d16: Status 404 returned error can't find the container with id 80f1bdd5a48e9c889a83a71399b182ddb67c1c6d675b97597809b8ba1ddc7d16 Jul 10 19:08:35.675098 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:35.675075 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77d7f68d48-227th"] Jul 10 19:08:35.676761 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:08:35.676735 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79af4c9e_1861_49bc_9d77_e07174ba3495.slice/crio-d3902db259d5310f18c4e07b3e896a952c30c8166810e55eac6553e7abf30692 WatchSource:0}: Error finding container d3902db259d5310f18c4e07b3e896a952c30c8166810e55eac6553e7abf30692: Status 404 returned error can't find the container with id d3902db259d5310f18c4e07b3e896a952c30c8166810e55eac6553e7abf30692 Jul 10 19:08:36.473046 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:36.473016 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69hnr_7f479cec-2fb8-45c4-ae7d-073194a5dede/ovn-acl-logging/0.log" Jul 10 19:08:36.478841 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:36.478811 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69hnr_7f479cec-2fb8-45c4-ae7d-073194a5dede/ovn-acl-logging/0.log" Jul 10 19:08:36.479862 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:36.479828 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77d7f68d48-227th" event={"ID":"79af4c9e-1861-49bc-9d77-e07174ba3495","Type":"ContainerStarted","Data":"d3902db259d5310f18c4e07b3e896a952c30c8166810e55eac6553e7abf30692"} Jul 10 19:08:36.483393 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:36.483362 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84f7ddd9bc-5xsvn" event={"ID":"35277ea5-79dc-4107-b8b3-ec94330189ef","Type":"ContainerStarted","Data":"80f1bdd5a48e9c889a83a71399b182ddb67c1c6d675b97597809b8ba1ddc7d16"} Jul 10 19:08:37.935551 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:37.935528 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 10 19:08:38.497305 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:38.497263 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84f7ddd9bc-5xsvn" event={"ID":"35277ea5-79dc-4107-b8b3-ec94330189ef","Type":"ContainerStarted","Data":"d16cc809c497665112ee70b84f91155095495d04cbbacd17a47d560b6fab78e4"} Jul 10 19:08:38.497496 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:38.497335 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-84f7ddd9bc-5xsvn" Jul 10 19:08:38.498869 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:38.498844 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77d7f68d48-227th" event={"ID":"79af4c9e-1861-49bc-9d77-e07174ba3495","Type":"ContainerStarted","Data":"cb7283fda3d49472e3c9b9c194ce8eda9831a08ea9f09b8725e88f1a2b5fb79d"} Jul 10 19:08:38.498989 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:38.498969 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-77d7f68d48-227th" Jul 10 19:08:38.531523 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:38.531472 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-84f7ddd9bc-5xsvn" podStartSLOduration=2.245685744 podStartE2EDuration="4.531454321s" totalImagesPullingTime="2.285768577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:08:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:08:38.528916479 +0000 UTC m=+602.573058981" watchObservedRunningTime="2026-07-10 19:08:38.531454321 +0000 UTC m=+602.575596843" Jul 10 19:08:38.564216 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:38.564160 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-77d7f68d48-227th" podStartSLOduration=1.298282834 podStartE2EDuration="3.564144482s" totalImagesPullingTime="2.265861648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:08:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:08:38.562748712 +0000 UTC m=+602.606891234" watchObservedRunningTime="2026-07-10 19:08:38.564144482 +0000 UTC m=+602.608286983" Jul 10 19:08:49.506173 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:49.506136 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-84f7ddd9bc-5xsvn" Jul 10 19:08:49.506566 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:49.506209 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-77d7f68d48-227th" Jul 10 19:08:49.572916 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:49.572880 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84f7ddd9bc-5xsvn"] Jul 10 19:08:49.573126 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:49.573103 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-84f7ddd9bc-5xsvn" podUID="35277ea5-79dc-4107-b8b3-ec94330189ef" containerName="manager" containerID="cri-o://d16cc809c497665112ee70b84f91155095495d04cbbacd17a47d560b6fab78e4" gracePeriod=30 Jul 10 19:08:49.823069 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:49.823042 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84f7ddd9bc-5xsvn" Jul 10 19:08:49.871478 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:49.871445 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xrlz\" (UniqueName: \"kubernetes.io/projected/35277ea5-79dc-4107-b8b3-ec94330189ef-kube-api-access-6xrlz\") pod \"35277ea5-79dc-4107-b8b3-ec94330189ef\" (UID: \"35277ea5-79dc-4107-b8b3-ec94330189ef\") " Jul 10 19:08:49.871633 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:49.871523 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/35277ea5-79dc-4107-b8b3-ec94330189ef-webhook-cert\") pod \"35277ea5-79dc-4107-b8b3-ec94330189ef\" (UID: \"35277ea5-79dc-4107-b8b3-ec94330189ef\") " Jul 10 19:08:49.873702 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:49.873666 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35277ea5-79dc-4107-b8b3-ec94330189ef-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "35277ea5-79dc-4107-b8b3-ec94330189ef" (UID: "35277ea5-79dc-4107-b8b3-ec94330189ef"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:08:49.873799 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:49.873667 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35277ea5-79dc-4107-b8b3-ec94330189ef-kube-api-access-6xrlz" (OuterVolumeSpecName: "kube-api-access-6xrlz") pod "35277ea5-79dc-4107-b8b3-ec94330189ef" (UID: "35277ea5-79dc-4107-b8b3-ec94330189ef"). InnerVolumeSpecName "kube-api-access-6xrlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:08:49.972249 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:49.972212 2581 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/35277ea5-79dc-4107-b8b3-ec94330189ef-webhook-cert\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:08:49.972249 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:49.972247 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xrlz\" (UniqueName: \"kubernetes.io/projected/35277ea5-79dc-4107-b8b3-ec94330189ef-kube-api-access-6xrlz\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:08:50.568761 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:50.568727 2581 generic.go:356] "Generic (PLEG): container finished" podID="35277ea5-79dc-4107-b8b3-ec94330189ef" containerID="d16cc809c497665112ee70b84f91155095495d04cbbacd17a47d560b6fab78e4" exitCode=0 Jul 10 19:08:50.569113 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:50.568771 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84f7ddd9bc-5xsvn" Jul 10 19:08:50.569113 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:50.568808 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84f7ddd9bc-5xsvn" event={"ID":"35277ea5-79dc-4107-b8b3-ec94330189ef","Type":"ContainerDied","Data":"d16cc809c497665112ee70b84f91155095495d04cbbacd17a47d560b6fab78e4"} Jul 10 19:08:50.569113 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:50.568860 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84f7ddd9bc-5xsvn" event={"ID":"35277ea5-79dc-4107-b8b3-ec94330189ef","Type":"ContainerDied","Data":"80f1bdd5a48e9c889a83a71399b182ddb67c1c6d675b97597809b8ba1ddc7d16"} Jul 10 19:08:50.569113 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:50.568883 2581 scope.go:117] "RemoveContainer" containerID="d16cc809c497665112ee70b84f91155095495d04cbbacd17a47d560b6fab78e4" Jul 10 19:08:50.584409 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:50.584390 2581 scope.go:117] "RemoveContainer" containerID="d16cc809c497665112ee70b84f91155095495d04cbbacd17a47d560b6fab78e4" Jul 10 19:08:50.584663 ip-10-0-135-67 kubenswrapper[2581]: E0710 19:08:50.584642 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d16cc809c497665112ee70b84f91155095495d04cbbacd17a47d560b6fab78e4\": container with ID starting with d16cc809c497665112ee70b84f91155095495d04cbbacd17a47d560b6fab78e4 not found: ID does not exist" containerID="d16cc809c497665112ee70b84f91155095495d04cbbacd17a47d560b6fab78e4" Jul 10 19:08:50.584748 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:50.584670 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d16cc809c497665112ee70b84f91155095495d04cbbacd17a47d560b6fab78e4"} err="failed to get container status \"d16cc809c497665112ee70b84f91155095495d04cbbacd17a47d560b6fab78e4\": rpc error: code = NotFound desc = could not find container \"d16cc809c497665112ee70b84f91155095495d04cbbacd17a47d560b6fab78e4\": container with ID starting with d16cc809c497665112ee70b84f91155095495d04cbbacd17a47d560b6fab78e4 not found: ID does not exist" Jul 10 19:08:50.590723 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:50.590701 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84f7ddd9bc-5xsvn"] Jul 10 19:08:50.594175 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:50.594153 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-84f7ddd9bc-5xsvn"] Jul 10 19:08:52.541840 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:08:52.541798 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35277ea5-79dc-4107-b8b3-ec94330189ef" path="/var/lib/kubelet/pods/35277ea5-79dc-4107-b8b3-ec94330189ef/volumes" Jul 10 19:09:11.486794 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:11.486717 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6789bd89c-sg8d5"] Jul 10 19:09:11.488396 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:11.487096 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35277ea5-79dc-4107-b8b3-ec94330189ef" containerName="manager" Jul 10 19:09:11.488396 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:11.487107 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="35277ea5-79dc-4107-b8b3-ec94330189ef" containerName="manager" Jul 10 19:09:11.488396 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:11.487175 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="35277ea5-79dc-4107-b8b3-ec94330189ef" containerName="manager" Jul 10 19:09:11.489062 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:11.489046 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6789bd89c-sg8d5" Jul 10 19:09:11.491447 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:11.491422 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 10 19:09:11.491665 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:11.491654 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-44lh8\"" Jul 10 19:09:11.502350 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:11.502321 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6789bd89c-sg8d5"] Jul 10 19:09:11.666234 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:11.666199 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p86kb\" (UniqueName: \"kubernetes.io/projected/143f3fb1-7601-4826-b1b6-6eddca57b03e-kube-api-access-p86kb\") pod \"maas-api-6789bd89c-sg8d5\" (UID: \"143f3fb1-7601-4826-b1b6-6eddca57b03e\") " pod="odh-ai-gateway-infra/maas-api-6789bd89c-sg8d5" Jul 10 19:09:11.666413 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:11.666254 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/143f3fb1-7601-4826-b1b6-6eddca57b03e-openshift-service-ca\") pod \"maas-api-6789bd89c-sg8d5\" (UID: \"143f3fb1-7601-4826-b1b6-6eddca57b03e\") " pod="odh-ai-gateway-infra/maas-api-6789bd89c-sg8d5" Jul 10 19:09:11.666413 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:11.666294 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/143f3fb1-7601-4826-b1b6-6eddca57b03e-maas-api-tls\") pod \"maas-api-6789bd89c-sg8d5\" (UID: \"143f3fb1-7601-4826-b1b6-6eddca57b03e\") " pod="odh-ai-gateway-infra/maas-api-6789bd89c-sg8d5" Jul 10 19:09:11.766729 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:11.766662 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/143f3fb1-7601-4826-b1b6-6eddca57b03e-maas-api-tls\") pod \"maas-api-6789bd89c-sg8d5\" (UID: \"143f3fb1-7601-4826-b1b6-6eddca57b03e\") " pod="odh-ai-gateway-infra/maas-api-6789bd89c-sg8d5" Jul 10 19:09:11.766897 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:11.766763 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p86kb\" (UniqueName: \"kubernetes.io/projected/143f3fb1-7601-4826-b1b6-6eddca57b03e-kube-api-access-p86kb\") pod \"maas-api-6789bd89c-sg8d5\" (UID: \"143f3fb1-7601-4826-b1b6-6eddca57b03e\") " pod="odh-ai-gateway-infra/maas-api-6789bd89c-sg8d5" Jul 10 19:09:11.766897 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:11.766810 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/143f3fb1-7601-4826-b1b6-6eddca57b03e-openshift-service-ca\") pod \"maas-api-6789bd89c-sg8d5\" (UID: \"143f3fb1-7601-4826-b1b6-6eddca57b03e\") " pod="odh-ai-gateway-infra/maas-api-6789bd89c-sg8d5" Jul 10 19:09:11.767524 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:11.767500 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/143f3fb1-7601-4826-b1b6-6eddca57b03e-openshift-service-ca\") pod \"maas-api-6789bd89c-sg8d5\" (UID: \"143f3fb1-7601-4826-b1b6-6eddca57b03e\") " pod="odh-ai-gateway-infra/maas-api-6789bd89c-sg8d5" Jul 10 19:09:11.769346 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:11.769325 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/143f3fb1-7601-4826-b1b6-6eddca57b03e-maas-api-tls\") pod \"maas-api-6789bd89c-sg8d5\" (UID: \"143f3fb1-7601-4826-b1b6-6eddca57b03e\") " pod="odh-ai-gateway-infra/maas-api-6789bd89c-sg8d5" Jul 10 19:09:11.774721 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:11.774673 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p86kb\" (UniqueName: \"kubernetes.io/projected/143f3fb1-7601-4826-b1b6-6eddca57b03e-kube-api-access-p86kb\") pod \"maas-api-6789bd89c-sg8d5\" (UID: \"143f3fb1-7601-4826-b1b6-6eddca57b03e\") " pod="odh-ai-gateway-infra/maas-api-6789bd89c-sg8d5" Jul 10 19:09:11.799486 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:11.799453 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6789bd89c-sg8d5" Jul 10 19:09:11.956078 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:11.956053 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6789bd89c-sg8d5"] Jul 10 19:09:11.957496 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:09:11.957467 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod143f3fb1_7601_4826_b1b6_6eddca57b03e.slice/crio-14ec0d584759f400dd0acf8012ffea3e46282d5d278165ecd9d2c97423736a2e WatchSource:0}: Error finding container 14ec0d584759f400dd0acf8012ffea3e46282d5d278165ecd9d2c97423736a2e: Status 404 returned error can't find the container with id 14ec0d584759f400dd0acf8012ffea3e46282d5d278165ecd9d2c97423736a2e Jul 10 19:09:11.958876 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:11.958852 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 19:09:12.693898 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:12.693857 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6789bd89c-sg8d5" event={"ID":"143f3fb1-7601-4826-b1b6-6eddca57b03e","Type":"ContainerStarted","Data":"14ec0d584759f400dd0acf8012ffea3e46282d5d278165ecd9d2c97423736a2e"} Jul 10 19:09:14.706889 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:14.706849 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6789bd89c-sg8d5" event={"ID":"143f3fb1-7601-4826-b1b6-6eddca57b03e","Type":"ContainerStarted","Data":"427fe1fb476f12d8f675957640813cf579956c6f33bfec31f4ce79bb1f5af230"} Jul 10 19:09:14.707292 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:14.706917 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6789bd89c-sg8d5" Jul 10 19:09:14.725087 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:14.725041 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6789bd89c-sg8d5" podStartSLOduration=1.760628576 podStartE2EDuration="3.725024425s" totalImagesPullingTime="1.964395849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:09:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:09:14.72357245 +0000 UTC m=+638.767714989" watchObservedRunningTime="2026-07-10 19:09:14.725024425 +0000 UTC m=+638.769166925" Jul 10 19:09:20.717879 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:20.717847 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6789bd89c-sg8d5" Jul 10 19:09:26.384533 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:26.384494 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v7n6p"] Jul 10 19:09:26.387930 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:26.387906 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v7n6p" Jul 10 19:09:26.391660 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:26.391634 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 10 19:09:26.391854 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:26.391834 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 10 19:09:26.391917 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:26.391835 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-plhxh\"" Jul 10 19:09:26.391917 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:26.391890 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 10 19:09:26.397326 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:26.397303 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v7n6p"] Jul 10 19:09:26.486628 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:26.486592 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2836da0c-b392-4604-a10b-13a9e7b489fe-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v7n6p\" (UID: \"2836da0c-b392-4604-a10b-13a9e7b489fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v7n6p" Jul 10 19:09:26.486826 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:26.486646 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2836da0c-b392-4604-a10b-13a9e7b489fe-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v7n6p\" (UID: \"2836da0c-b392-4604-a10b-13a9e7b489fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v7n6p" Jul 10 19:09:26.486826 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:26.486732 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2836da0c-b392-4604-a10b-13a9e7b489fe-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v7n6p\" (UID: \"2836da0c-b392-4604-a10b-13a9e7b489fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v7n6p" Jul 10 19:09:26.486826 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:26.486770 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2836da0c-b392-4604-a10b-13a9e7b489fe-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v7n6p\" (UID: \"2836da0c-b392-4604-a10b-13a9e7b489fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v7n6p" Jul 10 19:09:26.486826 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:26.486787 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2836da0c-b392-4604-a10b-13a9e7b489fe-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v7n6p\" (UID: \"2836da0c-b392-4604-a10b-13a9e7b489fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v7n6p" Jul 10 19:09:26.486826 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:26.486802 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng74j\" (UniqueName: \"kubernetes.io/projected/2836da0c-b392-4604-a10b-13a9e7b489fe-kube-api-access-ng74j\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v7n6p\" (UID: \"2836da0c-b392-4604-a10b-13a9e7b489fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v7n6p" Jul 10 19:09:26.587659 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:26.587621 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2836da0c-b392-4604-a10b-13a9e7b489fe-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v7n6p\" (UID: \"2836da0c-b392-4604-a10b-13a9e7b489fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v7n6p" Jul 10 19:09:26.587877 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:26.587668 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2836da0c-b392-4604-a10b-13a9e7b489fe-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v7n6p\" (UID: \"2836da0c-b392-4604-a10b-13a9e7b489fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v7n6p" Jul 10 19:09:26.587877 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:26.587765 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2836da0c-b392-4604-a10b-13a9e7b489fe-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v7n6p\" (UID: \"2836da0c-b392-4604-a10b-13a9e7b489fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v7n6p" Jul 10 19:09:26.587877 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:26.587800 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2836da0c-b392-4604-a10b-13a9e7b489fe-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v7n6p\" (UID: \"2836da0c-b392-4604-a10b-13a9e7b489fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v7n6p" Jul 10 19:09:26.587877 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:26.587830 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng74j\" (UniqueName: \"kubernetes.io/projected/2836da0c-b392-4604-a10b-13a9e7b489fe-kube-api-access-ng74j\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v7n6p\" (UID: \"2836da0c-b392-4604-a10b-13a9e7b489fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v7n6p" Jul 10 19:09:26.588097 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:26.587998 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2836da0c-b392-4604-a10b-13a9e7b489fe-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v7n6p\" (UID: \"2836da0c-b392-4604-a10b-13a9e7b489fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v7n6p" Jul 10 19:09:26.588236 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:26.588214 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2836da0c-b392-4604-a10b-13a9e7b489fe-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v7n6p\" (UID: \"2836da0c-b392-4604-a10b-13a9e7b489fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v7n6p" Jul 10 19:09:26.588289 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:26.588240 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2836da0c-b392-4604-a10b-13a9e7b489fe-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v7n6p\" (UID: \"2836da0c-b392-4604-a10b-13a9e7b489fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v7n6p" Jul 10 19:09:26.588375 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:26.588359 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2836da0c-b392-4604-a10b-13a9e7b489fe-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v7n6p\" (UID: \"2836da0c-b392-4604-a10b-13a9e7b489fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v7n6p" Jul 10 19:09:26.590091 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:26.590074 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2836da0c-b392-4604-a10b-13a9e7b489fe-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v7n6p\" (UID: \"2836da0c-b392-4604-a10b-13a9e7b489fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v7n6p" Jul 10 19:09:26.590193 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:26.590177 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2836da0c-b392-4604-a10b-13a9e7b489fe-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v7n6p\" (UID: \"2836da0c-b392-4604-a10b-13a9e7b489fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v7n6p" Jul 10 19:09:26.595971 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:26.595946 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng74j\" (UniqueName: \"kubernetes.io/projected/2836da0c-b392-4604-a10b-13a9e7b489fe-kube-api-access-ng74j\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v7n6p\" (UID: \"2836da0c-b392-4604-a10b-13a9e7b489fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v7n6p" Jul 10 19:09:26.698536 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:26.698449 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v7n6p" Jul 10 19:09:26.838740 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:26.838716 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v7n6p"] Jul 10 19:09:26.841118 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:09:26.841089 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2836da0c_b392_4604_a10b_13a9e7b489fe.slice/crio-2bcb8a81fb351046d32d5ced488b2722a71aae36217f7f79acd52298f964fb4b WatchSource:0}: Error finding container 2bcb8a81fb351046d32d5ced488b2722a71aae36217f7f79acd52298f964fb4b: Status 404 returned error can't find the container with id 2bcb8a81fb351046d32d5ced488b2722a71aae36217f7f79acd52298f964fb4b Jul 10 19:09:27.782627 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:27.782578 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v7n6p" event={"ID":"2836da0c-b392-4604-a10b-13a9e7b489fe","Type":"ContainerStarted","Data":"2bcb8a81fb351046d32d5ced488b2722a71aae36217f7f79acd52298f964fb4b"} Jul 10 19:09:28.789751 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:28.789711 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v7n6p" event={"ID":"2836da0c-b392-4604-a10b-13a9e7b489fe","Type":"ContainerStarted","Data":"1505f86a274cedd6c0cfc746e72e7c19e30de27fe13d73f3f3df7a286382a148"} Jul 10 19:09:28.808879 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:28.808804 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v7n6p" podStartSLOduration=1.7818986890000001 podStartE2EDuration="2.808789144s" totalImagesPullingTime="1.026890455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:09:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:09:28.807961712 +0000 UTC m=+652.852104214" watchObservedRunningTime="2026-07-10 19:09:28.808789144 +0000 UTC m=+652.852931644" Jul 10 19:09:36.698782 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:36.698740 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v7n6p" Jul 10 19:09:36.699189 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:36.698822 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v7n6p" Jul 10 19:09:36.711405 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:36.711376 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v7n6p" Jul 10 19:09:36.850640 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:36.850553 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v7n6p" Jul 10 19:09:59.803178 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:59.803139 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-zq4w6"] Jul 10 19:09:59.809270 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:59.809245 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zq4w6" Jul 10 19:09:59.811857 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:59.811828 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 10 19:09:59.820138 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:59.820113 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-zq4w6"] Jul 10 19:09:59.835749 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:59.835717 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63d0e5fe-2d13-40dd-9bbc-46dda9b19320-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zq4w6\" (UID: \"63d0e5fe-2d13-40dd-9bbc-46dda9b19320\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zq4w6" Jul 10 19:09:59.835877 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:59.835807 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63d0e5fe-2d13-40dd-9bbc-46dda9b19320-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zq4w6\" (UID: \"63d0e5fe-2d13-40dd-9bbc-46dda9b19320\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zq4w6" Jul 10 19:09:59.835933 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:59.835883 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63d0e5fe-2d13-40dd-9bbc-46dda9b19320-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zq4w6\" (UID: \"63d0e5fe-2d13-40dd-9bbc-46dda9b19320\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zq4w6" Jul 10 19:09:59.836014 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:59.835946 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bwc2\" (UniqueName: \"kubernetes.io/projected/63d0e5fe-2d13-40dd-9bbc-46dda9b19320-kube-api-access-4bwc2\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zq4w6\" (UID: \"63d0e5fe-2d13-40dd-9bbc-46dda9b19320\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zq4w6" Jul 10 19:09:59.836014 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:59.835971 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63d0e5fe-2d13-40dd-9bbc-46dda9b19320-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zq4w6\" (UID: \"63d0e5fe-2d13-40dd-9bbc-46dda9b19320\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zq4w6" Jul 10 19:09:59.836128 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:59.836079 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63d0e5fe-2d13-40dd-9bbc-46dda9b19320-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zq4w6\" (UID: \"63d0e5fe-2d13-40dd-9bbc-46dda9b19320\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zq4w6" Jul 10 19:09:59.937161 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:59.937126 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63d0e5fe-2d13-40dd-9bbc-46dda9b19320-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zq4w6\" (UID: \"63d0e5fe-2d13-40dd-9bbc-46dda9b19320\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zq4w6" Jul 10 19:09:59.937351 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:59.937177 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63d0e5fe-2d13-40dd-9bbc-46dda9b19320-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zq4w6\" (UID: \"63d0e5fe-2d13-40dd-9bbc-46dda9b19320\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zq4w6" Jul 10 19:09:59.937351 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:59.937298 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63d0e5fe-2d13-40dd-9bbc-46dda9b19320-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zq4w6\" (UID: \"63d0e5fe-2d13-40dd-9bbc-46dda9b19320\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zq4w6" Jul 10 19:09:59.937351 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:59.937339 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bwc2\" (UniqueName: \"kubernetes.io/projected/63d0e5fe-2d13-40dd-9bbc-46dda9b19320-kube-api-access-4bwc2\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zq4w6\" (UID: \"63d0e5fe-2d13-40dd-9bbc-46dda9b19320\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zq4w6" Jul 10 19:09:59.937550 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:59.937360 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63d0e5fe-2d13-40dd-9bbc-46dda9b19320-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zq4w6\" (UID: \"63d0e5fe-2d13-40dd-9bbc-46dda9b19320\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zq4w6" Jul 10 19:09:59.937550 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:59.937409 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63d0e5fe-2d13-40dd-9bbc-46dda9b19320-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zq4w6\" (UID: \"63d0e5fe-2d13-40dd-9bbc-46dda9b19320\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zq4w6" Jul 10 19:09:59.937701 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:59.937585 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63d0e5fe-2d13-40dd-9bbc-46dda9b19320-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zq4w6\" (UID: \"63d0e5fe-2d13-40dd-9bbc-46dda9b19320\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zq4w6" Jul 10 19:09:59.937701 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:59.937607 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63d0e5fe-2d13-40dd-9bbc-46dda9b19320-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zq4w6\" (UID: \"63d0e5fe-2d13-40dd-9bbc-46dda9b19320\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zq4w6" Jul 10 19:09:59.937852 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:59.937747 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63d0e5fe-2d13-40dd-9bbc-46dda9b19320-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zq4w6\" (UID: \"63d0e5fe-2d13-40dd-9bbc-46dda9b19320\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zq4w6" Jul 10 19:09:59.939746 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:59.939715 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63d0e5fe-2d13-40dd-9bbc-46dda9b19320-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zq4w6\" (UID: \"63d0e5fe-2d13-40dd-9bbc-46dda9b19320\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zq4w6" Jul 10 19:09:59.939871 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:59.939801 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63d0e5fe-2d13-40dd-9bbc-46dda9b19320-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zq4w6\" (UID: \"63d0e5fe-2d13-40dd-9bbc-46dda9b19320\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zq4w6" Jul 10 19:09:59.948145 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:09:59.948119 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bwc2\" (UniqueName: \"kubernetes.io/projected/63d0e5fe-2d13-40dd-9bbc-46dda9b19320-kube-api-access-4bwc2\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zq4w6\" (UID: \"63d0e5fe-2d13-40dd-9bbc-46dda9b19320\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zq4w6" Jul 10 19:10:00.122293 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:10:00.122200 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zq4w6" Jul 10 19:10:00.262745 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:10:00.262714 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-zq4w6"] Jul 10 19:10:00.264366 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:10:00.264340 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63d0e5fe_2d13_40dd_9bbc_46dda9b19320.slice/crio-2068ad96dd8bac94936253b128838a2f25f37e140c1a134c3fd22db4ee03d833 WatchSource:0}: Error finding container 2068ad96dd8bac94936253b128838a2f25f37e140c1a134c3fd22db4ee03d833: Status 404 returned error can't find the container with id 2068ad96dd8bac94936253b128838a2f25f37e140c1a134c3fd22db4ee03d833 Jul 10 19:10:00.976392 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:10:00.976309 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zq4w6" event={"ID":"63d0e5fe-2d13-40dd-9bbc-46dda9b19320","Type":"ContainerStarted","Data":"7f42f90d88626c6c887da70a3e768e66374af98ef6f9a080b2f43ffb8342133c"} Jul 10 19:10:00.976392 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:10:00.976358 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zq4w6" event={"ID":"63d0e5fe-2d13-40dd-9bbc-46dda9b19320","Type":"ContainerStarted","Data":"2068ad96dd8bac94936253b128838a2f25f37e140c1a134c3fd22db4ee03d833"} Jul 10 19:10:00.996808 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:10:00.996757 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zq4w6" podStartSLOduration=1.721448485 podStartE2EDuration="1.996740451s" totalImagesPullingTime="275.291966ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:09:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:10:00.99430254 +0000 UTC m=+685.038445063" watchObservedRunningTime="2026-07-10 19:10:00.996740451 +0000 UTC m=+685.040882952" Jul 10 19:10:10.122490 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:10:10.122450 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zq4w6" Jul 10 19:10:10.123028 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:10:10.122522 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zq4w6" Jul 10 19:10:10.138742 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:10:10.138709 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zq4w6" Jul 10 19:10:11.048849 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:10:11.048820 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zq4w6" Jul 10 19:10:20.095717 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:10:20.094230 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-g22tg"] Jul 10 19:10:20.102219 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:10:20.102191 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g22tg" Jul 10 19:10:20.105075 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:10:20.105044 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 10 19:10:20.107927 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:10:20.107898 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-g22tg"] Jul 10 19:10:20.216303 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:10:20.216272 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnvcd\" (UniqueName: \"kubernetes.io/projected/da3fe636-22e0-4a46-b9a5-49653ba2b691-kube-api-access-xnvcd\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g22tg\" (UID: \"da3fe636-22e0-4a46-b9a5-49653ba2b691\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g22tg" Jul 10 19:10:20.216303 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:10:20.216307 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da3fe636-22e0-4a46-b9a5-49653ba2b691-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g22tg\" (UID: \"da3fe636-22e0-4a46-b9a5-49653ba2b691\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g22tg" Jul 10 19:10:20.216537 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:10:20.216329 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da3fe636-22e0-4a46-b9a5-49653ba2b691-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g22tg\" (UID: \"da3fe636-22e0-4a46-b9a5-49653ba2b691\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g22tg" Jul 10 19:10:20.216537 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:10:20.216417 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da3fe636-22e0-4a46-b9a5-49653ba2b691-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g22tg\" (UID: \"da3fe636-22e0-4a46-b9a5-49653ba2b691\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g22tg" Jul 10 19:10:20.216537 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:10:20.216449 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da3fe636-22e0-4a46-b9a5-49653ba2b691-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g22tg\" (UID: \"da3fe636-22e0-4a46-b9a5-49653ba2b691\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g22tg" Jul 10 19:10:20.216537 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:10:20.216487 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da3fe636-22e0-4a46-b9a5-49653ba2b691-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g22tg\" (UID: \"da3fe636-22e0-4a46-b9a5-49653ba2b691\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g22tg" Jul 10 19:10:20.317796 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:10:20.317758 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnvcd\" (UniqueName: \"kubernetes.io/projected/da3fe636-22e0-4a46-b9a5-49653ba2b691-kube-api-access-xnvcd\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g22tg\" (UID: \"da3fe636-22e0-4a46-b9a5-49653ba2b691\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g22tg" Jul 10 19:10:20.317988 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:10:20.317804 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da3fe636-22e0-4a46-b9a5-49653ba2b691-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g22tg\" (UID: \"da3fe636-22e0-4a46-b9a5-49653ba2b691\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g22tg" Jul 10 19:10:20.317988 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:10:20.317833 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da3fe636-22e0-4a46-b9a5-49653ba2b691-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g22tg\" (UID: \"da3fe636-22e0-4a46-b9a5-49653ba2b691\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g22tg" Jul 10 19:10:20.317988 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:10:20.317884 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da3fe636-22e0-4a46-b9a5-49653ba2b691-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g22tg\" (UID: \"da3fe636-22e0-4a46-b9a5-49653ba2b691\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g22tg" Jul 10 19:10:20.317988 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:10:20.317913 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da3fe636-22e0-4a46-b9a5-49653ba2b691-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g22tg\" (UID: \"da3fe636-22e0-4a46-b9a5-49653ba2b691\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g22tg" Jul 10 19:10:20.317988 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:10:20.317963 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da3fe636-22e0-4a46-b9a5-49653ba2b691-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g22tg\" (UID: \"da3fe636-22e0-4a46-b9a5-49653ba2b691\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g22tg" Jul 10 19:10:20.318319 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:10:20.318298 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da3fe636-22e0-4a46-b9a5-49653ba2b691-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g22tg\" (UID: \"da3fe636-22e0-4a46-b9a5-49653ba2b691\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g22tg" Jul 10 19:10:20.318359 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:10:20.318341 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da3fe636-22e0-4a46-b9a5-49653ba2b691-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g22tg\" (UID: \"da3fe636-22e0-4a46-b9a5-49653ba2b691\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g22tg" Jul 10 19:10:20.318391 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:10:20.318363 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da3fe636-22e0-4a46-b9a5-49653ba2b691-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g22tg\" (UID: \"da3fe636-22e0-4a46-b9a5-49653ba2b691\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g22tg" Jul 10 19:10:20.320216 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:10:20.320193 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da3fe636-22e0-4a46-b9a5-49653ba2b691-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g22tg\" (UID: \"da3fe636-22e0-4a46-b9a5-49653ba2b691\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g22tg" Jul 10 19:10:20.320638 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:10:20.320611 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da3fe636-22e0-4a46-b9a5-49653ba2b691-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g22tg\" (UID: \"da3fe636-22e0-4a46-b9a5-49653ba2b691\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g22tg" Jul 10 19:10:20.326113 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:10:20.326091 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnvcd\" (UniqueName: \"kubernetes.io/projected/da3fe636-22e0-4a46-b9a5-49653ba2b691-kube-api-access-xnvcd\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-g22tg\" (UID: \"da3fe636-22e0-4a46-b9a5-49653ba2b691\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g22tg" Jul 10 19:10:20.414170 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:10:20.414077 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g22tg" Jul 10 19:10:20.560125 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:10:20.560097 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-g22tg"] Jul 10 19:10:20.561834 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:10:20.561808 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda3fe636_22e0_4a46_b9a5_49653ba2b691.slice/crio-d0dbe9e13d41f6752f147c2d11bdb15f66987335392304630aa06d25a3766870 WatchSource:0}: Error finding container d0dbe9e13d41f6752f147c2d11bdb15f66987335392304630aa06d25a3766870: Status 404 returned error can't find the container with id d0dbe9e13d41f6752f147c2d11bdb15f66987335392304630aa06d25a3766870 Jul 10 19:10:21.093123 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:10:21.093028 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g22tg" event={"ID":"da3fe636-22e0-4a46-b9a5-49653ba2b691","Type":"ContainerStarted","Data":"f1e320d2e46d9a2910028d12a006464120122777b2f92f2eb81d7646aad32bd8"} Jul 10 19:10:21.093123 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:10:21.093093 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g22tg" event={"ID":"da3fe636-22e0-4a46-b9a5-49653ba2b691","Type":"ContainerStarted","Data":"d0dbe9e13d41f6752f147c2d11bdb15f66987335392304630aa06d25a3766870"} Jul 10 19:10:21.112981 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:10:21.112934 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g22tg" podStartSLOduration=0.924280689 podStartE2EDuration="1.112918627s" totalImagesPullingTime="188.637938ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:10:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:10:21.11092719 +0000 UTC m=+705.155069692" watchObservedRunningTime="2026-07-10 19:10:21.112918627 +0000 UTC m=+705.157061132" Jul 10 19:10:30.414265 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:10:30.414181 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g22tg" Jul 10 19:10:30.414265 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:10:30.414256 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g22tg" Jul 10 19:10:30.426759 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:10:30.426733 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g22tg" Jul 10 19:10:31.159677 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:10:31.159647 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-g22tg" Jul 10 19:12:24.189084 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:12:24.188993 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-77d7f68d48-227th"] Jul 10 19:12:24.189629 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:12:24.189274 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-77d7f68d48-227th" podUID="79af4c9e-1861-49bc-9d77-e07174ba3495" containerName="manager" containerID="cri-o://cb7283fda3d49472e3c9b9c194ce8eda9831a08ea9f09b8725e88f1a2b5fb79d" gracePeriod=30 Jul 10 19:12:24.447565 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:12:24.447493 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77d7f68d48-227th" Jul 10 19:12:24.618670 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:12:24.618634 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/79af4c9e-1861-49bc-9d77-e07174ba3495-webhook-cert\") pod \"79af4c9e-1861-49bc-9d77-e07174ba3495\" (UID: \"79af4c9e-1861-49bc-9d77-e07174ba3495\") " Jul 10 19:12:24.618670 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:12:24.618672 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vpn6m\" (UniqueName: \"kubernetes.io/projected/79af4c9e-1861-49bc-9d77-e07174ba3495-kube-api-access-vpn6m\") pod \"79af4c9e-1861-49bc-9d77-e07174ba3495\" (UID: \"79af4c9e-1861-49bc-9d77-e07174ba3495\") " Jul 10 19:12:24.620677 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:12:24.620649 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79af4c9e-1861-49bc-9d77-e07174ba3495-kube-api-access-vpn6m" (OuterVolumeSpecName: "kube-api-access-vpn6m") pod "79af4c9e-1861-49bc-9d77-e07174ba3495" (UID: "79af4c9e-1861-49bc-9d77-e07174ba3495"). InnerVolumeSpecName "kube-api-access-vpn6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:12:24.620807 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:12:24.620766 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79af4c9e-1861-49bc-9d77-e07174ba3495-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "79af4c9e-1861-49bc-9d77-e07174ba3495" (UID: "79af4c9e-1861-49bc-9d77-e07174ba3495"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:12:24.720306 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:12:24.720217 2581 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/79af4c9e-1861-49bc-9d77-e07174ba3495-webhook-cert\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:12:24.720306 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:12:24.720250 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpn6m\" (UniqueName: \"kubernetes.io/projected/79af4c9e-1861-49bc-9d77-e07174ba3495-kube-api-access-vpn6m\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:12:24.777352 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:12:24.777311 2581 generic.go:356] "Generic (PLEG): container finished" podID="79af4c9e-1861-49bc-9d77-e07174ba3495" containerID="cb7283fda3d49472e3c9b9c194ce8eda9831a08ea9f09b8725e88f1a2b5fb79d" exitCode=0 Jul 10 19:12:24.777535 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:12:24.777360 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77d7f68d48-227th" Jul 10 19:12:24.777535 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:12:24.777393 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77d7f68d48-227th" event={"ID":"79af4c9e-1861-49bc-9d77-e07174ba3495","Type":"ContainerDied","Data":"cb7283fda3d49472e3c9b9c194ce8eda9831a08ea9f09b8725e88f1a2b5fb79d"} Jul 10 19:12:24.777535 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:12:24.777443 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77d7f68d48-227th" event={"ID":"79af4c9e-1861-49bc-9d77-e07174ba3495","Type":"ContainerDied","Data":"d3902db259d5310f18c4e07b3e896a952c30c8166810e55eac6553e7abf30692"} Jul 10 19:12:24.777535 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:12:24.777466 2581 scope.go:117] "RemoveContainer" containerID="cb7283fda3d49472e3c9b9c194ce8eda9831a08ea9f09b8725e88f1a2b5fb79d" Jul 10 19:12:24.787161 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:12:24.787143 2581 scope.go:117] "RemoveContainer" containerID="cb7283fda3d49472e3c9b9c194ce8eda9831a08ea9f09b8725e88f1a2b5fb79d" Jul 10 19:12:24.787478 ip-10-0-135-67 kubenswrapper[2581]: E0710 19:12:24.787455 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb7283fda3d49472e3c9b9c194ce8eda9831a08ea9f09b8725e88f1a2b5fb79d\": container with ID starting with cb7283fda3d49472e3c9b9c194ce8eda9831a08ea9f09b8725e88f1a2b5fb79d not found: ID does not exist" containerID="cb7283fda3d49472e3c9b9c194ce8eda9831a08ea9f09b8725e88f1a2b5fb79d" Jul 10 19:12:24.787548 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:12:24.787494 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb7283fda3d49472e3c9b9c194ce8eda9831a08ea9f09b8725e88f1a2b5fb79d"} err="failed to get container status \"cb7283fda3d49472e3c9b9c194ce8eda9831a08ea9f09b8725e88f1a2b5fb79d\": rpc error: code = NotFound desc = could not find container \"cb7283fda3d49472e3c9b9c194ce8eda9831a08ea9f09b8725e88f1a2b5fb79d\": container with ID starting with cb7283fda3d49472e3c9b9c194ce8eda9831a08ea9f09b8725e88f1a2b5fb79d not found: ID does not exist" Jul 10 19:12:24.799849 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:12:24.799817 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-77d7f68d48-227th"] Jul 10 19:12:24.803323 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:12:24.803300 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-77d7f68d48-227th"] Jul 10 19:12:25.542698 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:12:25.542652 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-77d7f68d48-hnrxv"] Jul 10 19:12:25.543125 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:12:25.543111 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79af4c9e-1861-49bc-9d77-e07174ba3495" containerName="manager" Jul 10 19:12:25.543168 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:12:25.543126 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="79af4c9e-1861-49bc-9d77-e07174ba3495" containerName="manager" Jul 10 19:12:25.543202 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:12:25.543194 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="79af4c9e-1861-49bc-9d77-e07174ba3495" containerName="manager" Jul 10 19:12:25.547758 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:12:25.547736 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77d7f68d48-hnrxv" Jul 10 19:12:25.551015 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:12:25.550988 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 10 19:12:25.551150 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:12:25.550989 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-4b97p\"" Jul 10 19:12:25.551150 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:12:25.551032 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 10 19:12:25.555282 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:12:25.555068 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77d7f68d48-hnrxv"] Jul 10 19:12:25.629022 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:12:25.628982 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd7nv\" (UniqueName: \"kubernetes.io/projected/2e1f0282-7303-43da-ae3c-767a14ab8c16-kube-api-access-bd7nv\") pod \"maas-controller-77d7f68d48-hnrxv\" (UID: \"2e1f0282-7303-43da-ae3c-767a14ab8c16\") " pod="opendatahub/maas-controller-77d7f68d48-hnrxv" Jul 10 19:12:25.629221 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:12:25.629055 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e1f0282-7303-43da-ae3c-767a14ab8c16-webhook-cert\") pod \"maas-controller-77d7f68d48-hnrxv\" (UID: \"2e1f0282-7303-43da-ae3c-767a14ab8c16\") " pod="opendatahub/maas-controller-77d7f68d48-hnrxv" Jul 10 19:12:25.729921 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:12:25.729885 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd7nv\" (UniqueName: \"kubernetes.io/projected/2e1f0282-7303-43da-ae3c-767a14ab8c16-kube-api-access-bd7nv\") pod \"maas-controller-77d7f68d48-hnrxv\" (UID: \"2e1f0282-7303-43da-ae3c-767a14ab8c16\") " pod="opendatahub/maas-controller-77d7f68d48-hnrxv" Jul 10 19:12:25.730110 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:12:25.729959 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e1f0282-7303-43da-ae3c-767a14ab8c16-webhook-cert\") pod \"maas-controller-77d7f68d48-hnrxv\" (UID: \"2e1f0282-7303-43da-ae3c-767a14ab8c16\") " pod="opendatahub/maas-controller-77d7f68d48-hnrxv" Jul 10 19:12:25.732729 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:12:25.732672 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e1f0282-7303-43da-ae3c-767a14ab8c16-webhook-cert\") pod \"maas-controller-77d7f68d48-hnrxv\" (UID: \"2e1f0282-7303-43da-ae3c-767a14ab8c16\") " pod="opendatahub/maas-controller-77d7f68d48-hnrxv" Jul 10 19:12:25.738789 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:12:25.738759 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd7nv\" (UniqueName: \"kubernetes.io/projected/2e1f0282-7303-43da-ae3c-767a14ab8c16-kube-api-access-bd7nv\") pod \"maas-controller-77d7f68d48-hnrxv\" (UID: \"2e1f0282-7303-43da-ae3c-767a14ab8c16\") " pod="opendatahub/maas-controller-77d7f68d48-hnrxv" Jul 10 19:12:25.859495 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:12:25.859396 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77d7f68d48-hnrxv" Jul 10 19:12:25.994644 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:12:25.994620 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77d7f68d48-hnrxv"] Jul 10 19:12:25.996744 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:12:25.996664 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e1f0282_7303_43da_ae3c_767a14ab8c16.slice/crio-a77165092211d7b5e354c246f2f200dfdf3b94cf9c38cd3fbdfb2834e573e687 WatchSource:0}: Error finding container a77165092211d7b5e354c246f2f200dfdf3b94cf9c38cd3fbdfb2834e573e687: Status 404 returned error can't find the container with id a77165092211d7b5e354c246f2f200dfdf3b94cf9c38cd3fbdfb2834e573e687 Jul 10 19:12:26.542595 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:12:26.542566 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79af4c9e-1861-49bc-9d77-e07174ba3495" path="/var/lib/kubelet/pods/79af4c9e-1861-49bc-9d77-e07174ba3495/volumes" Jul 10 19:12:26.790421 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:12:26.790320 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77d7f68d48-hnrxv" event={"ID":"2e1f0282-7303-43da-ae3c-767a14ab8c16","Type":"ContainerStarted","Data":"652aaf51f6d250d2e6c041c58e36e44959125aedb65f459d22201baee7855645"} Jul 10 19:12:26.790421 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:12:26.790386 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77d7f68d48-hnrxv" event={"ID":"2e1f0282-7303-43da-ae3c-767a14ab8c16","Type":"ContainerStarted","Data":"a77165092211d7b5e354c246f2f200dfdf3b94cf9c38cd3fbdfb2834e573e687"} Jul 10 19:12:26.790421 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:12:26.790414 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-77d7f68d48-hnrxv" Jul 10 19:12:26.808337 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:12:26.808283 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-77d7f68d48-hnrxv" podStartSLOduration=1.2936731639999999 podStartE2EDuration="1.80826629s" totalImagesPullingTime="514.593126ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:12:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:12:26.807119578 +0000 UTC m=+830.851262081" watchObservedRunningTime="2026-07-10 19:12:26.80826629 +0000 UTC m=+830.852408790" Jul 10 19:12:37.798818 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:12:37.798786 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-77d7f68d48-hnrxv" Jul 10 19:13:36.519127 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:13:36.519100 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69hnr_7f479cec-2fb8-45c4-ae7d-073194a5dede/ovn-acl-logging/0.log" Jul 10 19:13:36.526821 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:13:36.526799 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69hnr_7f479cec-2fb8-45c4-ae7d-073194a5dede/ovn-acl-logging/0.log" Jul 10 19:17:11.862672 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:17:11.862633 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wqfnm"] Jul 10 19:17:11.863244 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:17:11.862877 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wqfnm" podUID="d7981090-2b0a-4f1f-949b-218b720d5f7c" containerName="manager" containerID="cri-o://d752d29f3c470d13c3b8c4c0566d4e71a30eec74afe2089bfb890be11ac60ede" gracePeriod=10 Jul 10 19:17:15.131287 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:17:15.131258 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wqfnm" Jul 10 19:17:15.209981 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:17:15.209897 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t29wp\" (UniqueName: \"kubernetes.io/projected/d7981090-2b0a-4f1f-949b-218b720d5f7c-kube-api-access-t29wp\") pod \"d7981090-2b0a-4f1f-949b-218b720d5f7c\" (UID: \"d7981090-2b0a-4f1f-949b-218b720d5f7c\") " Jul 10 19:17:15.210117 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:17:15.209994 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d7981090-2b0a-4f1f-949b-218b720d5f7c-extensions-socket-volume\") pod \"d7981090-2b0a-4f1f-949b-218b720d5f7c\" (UID: \"d7981090-2b0a-4f1f-949b-218b720d5f7c\") " Jul 10 19:17:15.210386 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:17:15.210359 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7981090-2b0a-4f1f-949b-218b720d5f7c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d7981090-2b0a-4f1f-949b-218b720d5f7c" (UID: "d7981090-2b0a-4f1f-949b-218b720d5f7c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:17:15.211946 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:17:15.211924 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7981090-2b0a-4f1f-949b-218b720d5f7c-kube-api-access-t29wp" (OuterVolumeSpecName: "kube-api-access-t29wp") pod "d7981090-2b0a-4f1f-949b-218b720d5f7c" (UID: "d7981090-2b0a-4f1f-949b-218b720d5f7c"). InnerVolumeSpecName "kube-api-access-t29wp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:17:15.310654 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:17:15.310613 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t29wp\" (UniqueName: \"kubernetes.io/projected/d7981090-2b0a-4f1f-949b-218b720d5f7c-kube-api-access-t29wp\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:17:15.310654 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:17:15.310648 2581 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d7981090-2b0a-4f1f-949b-218b720d5f7c-extensions-socket-volume\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:17:15.394539 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:17:15.394505 2581 generic.go:356] "Generic (PLEG): container finished" podID="d7981090-2b0a-4f1f-949b-218b720d5f7c" containerID="d752d29f3c470d13c3b8c4c0566d4e71a30eec74afe2089bfb890be11ac60ede" exitCode=0 Jul 10 19:17:15.394706 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:17:15.394570 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wqfnm" Jul 10 19:17:15.394706 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:17:15.394601 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wqfnm" event={"ID":"d7981090-2b0a-4f1f-949b-218b720d5f7c","Type":"ContainerDied","Data":"d752d29f3c470d13c3b8c4c0566d4e71a30eec74afe2089bfb890be11ac60ede"} Jul 10 19:17:15.394706 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:17:15.394675 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wqfnm" event={"ID":"d7981090-2b0a-4f1f-949b-218b720d5f7c","Type":"ContainerDied","Data":"130ff94986a428eb896f06fdd195fd2b53e23e7a3a9ef4bc6d3d779ca63a4eb4"} Jul 10 19:17:15.394838 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:17:15.394715 2581 scope.go:117] "RemoveContainer" containerID="d752d29f3c470d13c3b8c4c0566d4e71a30eec74afe2089bfb890be11ac60ede" Jul 10 19:17:15.405040 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:17:15.405017 2581 scope.go:117] "RemoveContainer" containerID="d752d29f3c470d13c3b8c4c0566d4e71a30eec74afe2089bfb890be11ac60ede" Jul 10 19:17:15.405306 ip-10-0-135-67 kubenswrapper[2581]: E0710 19:17:15.405287 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d752d29f3c470d13c3b8c4c0566d4e71a30eec74afe2089bfb890be11ac60ede\": container with ID starting with d752d29f3c470d13c3b8c4c0566d4e71a30eec74afe2089bfb890be11ac60ede not found: ID does not exist" containerID="d752d29f3c470d13c3b8c4c0566d4e71a30eec74afe2089bfb890be11ac60ede" Jul 10 19:17:15.405352 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:17:15.405315 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d752d29f3c470d13c3b8c4c0566d4e71a30eec74afe2089bfb890be11ac60ede"} err="failed to get container status \"d752d29f3c470d13c3b8c4c0566d4e71a30eec74afe2089bfb890be11ac60ede\": rpc error: code = NotFound desc = could not find container \"d752d29f3c470d13c3b8c4c0566d4e71a30eec74afe2089bfb890be11ac60ede\": container with ID starting with d752d29f3c470d13c3b8c4c0566d4e71a30eec74afe2089bfb890be11ac60ede not found: ID does not exist" Jul 10 19:17:15.416576 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:17:15.416549 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wqfnm"] Jul 10 19:17:15.423315 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:17:15.423287 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wqfnm"] Jul 10 19:17:16.544357 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:17:16.544324 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7981090-2b0a-4f1f-949b-218b720d5f7c" path="/var/lib/kubelet/pods/d7981090-2b0a-4f1f-949b-218b720d5f7c/volumes" Jul 10 19:18:12.959964 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:18:12.959880 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6pxgk"] Jul 10 19:18:12.960443 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:18:12.960324 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7981090-2b0a-4f1f-949b-218b720d5f7c" containerName="manager" Jul 10 19:18:12.960443 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:18:12.960337 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7981090-2b0a-4f1f-949b-218b720d5f7c" containerName="manager" Jul 10 19:18:12.960443 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:18:12.960404 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7981090-2b0a-4f1f-949b-218b720d5f7c" containerName="manager" Jul 10 19:18:12.963480 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:18:12.963463 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6pxgk" Jul 10 19:18:12.966101 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:18:12.966079 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-lt9xj\"" Jul 10 19:18:12.980469 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:18:12.980441 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6pxgk"] Jul 10 19:18:13.008234 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:18:13.008200 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/88426ec8-8c7e-4f16-b4c8-7ba42a36d8ac-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-6pxgk\" (UID: \"88426ec8-8c7e-4f16-b4c8-7ba42a36d8ac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6pxgk" Jul 10 19:18:13.008444 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:18:13.008364 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5rvc\" (UniqueName: \"kubernetes.io/projected/88426ec8-8c7e-4f16-b4c8-7ba42a36d8ac-kube-api-access-b5rvc\") pod \"kuadrant-operator-controller-manager-86878c769f-6pxgk\" (UID: \"88426ec8-8c7e-4f16-b4c8-7ba42a36d8ac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6pxgk" Jul 10 19:18:13.109548 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:18:13.109506 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5rvc\" (UniqueName: \"kubernetes.io/projected/88426ec8-8c7e-4f16-b4c8-7ba42a36d8ac-kube-api-access-b5rvc\") pod \"kuadrant-operator-controller-manager-86878c769f-6pxgk\" (UID: \"88426ec8-8c7e-4f16-b4c8-7ba42a36d8ac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6pxgk" Jul 10 19:18:13.109548 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:18:13.109551 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/88426ec8-8c7e-4f16-b4c8-7ba42a36d8ac-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-6pxgk\" (UID: \"88426ec8-8c7e-4f16-b4c8-7ba42a36d8ac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6pxgk" Jul 10 19:18:13.109996 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:18:13.109978 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/88426ec8-8c7e-4f16-b4c8-7ba42a36d8ac-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-6pxgk\" (UID: \"88426ec8-8c7e-4f16-b4c8-7ba42a36d8ac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6pxgk" Jul 10 19:18:13.120934 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:18:13.120905 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5rvc\" (UniqueName: \"kubernetes.io/projected/88426ec8-8c7e-4f16-b4c8-7ba42a36d8ac-kube-api-access-b5rvc\") pod \"kuadrant-operator-controller-manager-86878c769f-6pxgk\" (UID: \"88426ec8-8c7e-4f16-b4c8-7ba42a36d8ac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6pxgk" Jul 10 19:18:13.274357 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:18:13.274319 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6pxgk" Jul 10 19:18:13.420125 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:18:13.420095 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6pxgk"] Jul 10 19:18:13.422809 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:18:13.422784 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88426ec8_8c7e_4f16_b4c8_7ba42a36d8ac.slice/crio-c19338bbb04ee398618c07b53ab81f727fc0c33a01a266741f19929b4d97bc1f WatchSource:0}: Error finding container c19338bbb04ee398618c07b53ab81f727fc0c33a01a266741f19929b4d97bc1f: Status 404 returned error can't find the container with id c19338bbb04ee398618c07b53ab81f727fc0c33a01a266741f19929b4d97bc1f Jul 10 19:18:13.723737 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:18:13.723622 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6pxgk" event={"ID":"88426ec8-8c7e-4f16-b4c8-7ba42a36d8ac","Type":"ContainerStarted","Data":"2da24306e4f2cb0d2bcc584d7cb5474a63bb380bc41dda53c918c728ab195d75"} Jul 10 19:18:13.723737 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:18:13.723711 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6pxgk" event={"ID":"88426ec8-8c7e-4f16-b4c8-7ba42a36d8ac","Type":"ContainerStarted","Data":"c19338bbb04ee398618c07b53ab81f727fc0c33a01a266741f19929b4d97bc1f"} Jul 10 19:18:13.723926 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:18:13.723742 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6pxgk" Jul 10 19:18:13.746669 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:18:13.746615 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6pxgk" podStartSLOduration=1.746598254 podStartE2EDuration="1.746598254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:18:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:18:13.743560623 +0000 UTC m=+1177.787703125" watchObservedRunningTime="2026-07-10 19:18:13.746598254 +0000 UTC m=+1177.790740809" Jul 10 19:18:24.732598 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:18:24.732567 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6pxgk" Jul 10 19:18:36.557292 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:18:36.557262 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69hnr_7f479cec-2fb8-45c4-ae7d-073194a5dede/ovn-acl-logging/0.log" Jul 10 19:18:36.565610 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:18:36.565586 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69hnr_7f479cec-2fb8-45c4-ae7d-073194a5dede/ovn-acl-logging/0.log" Jul 10 19:23:36.596079 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:23:36.596046 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69hnr_7f479cec-2fb8-45c4-ae7d-073194a5dede/ovn-acl-logging/0.log" Jul 10 19:23:36.605434 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:23:36.605410 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69hnr_7f479cec-2fb8-45c4-ae7d-073194a5dede/ovn-acl-logging/0.log" Jul 10 19:24:33.664474 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:24:33.664434 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e61e5c8-wg2pz"] Jul 10 19:24:33.667786 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:24:33.667768 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e61e5c8-wg2pz" Jul 10 19:24:33.670537 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:24:33.670514 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fsnnw\"" Jul 10 19:24:33.676317 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:24:33.676292 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e61e5c8-wg2pz"] Jul 10 19:24:33.699489 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:24:33.699452 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dbf6ad5-f44f-4f7b-b49b-5c561cdb4e4a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6e61e5c8-wg2pz\" (UID: \"7dbf6ad5-f44f-4f7b-b49b-5c561cdb4e4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e61e5c8-wg2pz" Jul 10 19:24:33.800830 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:24:33.800783 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dbf6ad5-f44f-4f7b-b49b-5c561cdb4e4a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6e61e5c8-wg2pz\" (UID: \"7dbf6ad5-f44f-4f7b-b49b-5c561cdb4e4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e61e5c8-wg2pz" Jul 10 19:24:33.801445 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:24:33.801422 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dbf6ad5-f44f-4f7b-b49b-5c561cdb4e4a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6e61e5c8-wg2pz\" (UID: \"7dbf6ad5-f44f-4f7b-b49b-5c561cdb4e4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e61e5c8-wg2pz" Jul 10 19:24:33.980860 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:24:33.980766 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e61e5c8-wg2pz" Jul 10 19:24:34.327563 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:24:34.327535 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e61e5c8-wg2pz"] Jul 10 19:24:34.329607 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:24:34.329577 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dbf6ad5_f44f_4f7b_b49b_5c561cdb4e4a.slice/crio-f287e1ab89d39333126c05808b0706dde6f85e5657eedaf77f5305cb6d749ce6 WatchSource:0}: Error finding container f287e1ab89d39333126c05808b0706dde6f85e5657eedaf77f5305cb6d749ce6: Status 404 returned error can't find the container with id f287e1ab89d39333126c05808b0706dde6f85e5657eedaf77f5305cb6d749ce6 Jul 10 19:24:34.332195 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:24:34.332166 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 19:24:34.880780 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:24:34.880745 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e61e5c8-wg2pz" event={"ID":"7dbf6ad5-f44f-4f7b-b49b-5c561cdb4e4a","Type":"ContainerStarted","Data":"f287e1ab89d39333126c05808b0706dde6f85e5657eedaf77f5305cb6d749ce6"} Jul 10 19:24:36.894374 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:24:36.894337 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e61e5c8-wg2pz" event={"ID":"7dbf6ad5-f44f-4f7b-b49b-5c561cdb4e4a","Type":"ContainerStarted","Data":"4f7be174ecec7d1529590f28f6adacd232c18394ee05792ee3f20a86ff56216f"} Jul 10 19:24:36.913417 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:24:36.913362 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e61e5c8-wg2pz" podStartSLOduration=1.7378731219999999 podStartE2EDuration="3.913347198s" totalImagesPullingTime="2.175474076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:24:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:24:36.910677088 +0000 UTC m=+1560.954819589" watchObservedRunningTime="2026-07-10 19:24:36.913347198 +0000 UTC m=+1560.957489699" Jul 10 19:24:37.901537 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:24:37.901501 2581 generic.go:356] "Generic (PLEG): container finished" podID="7dbf6ad5-f44f-4f7b-b49b-5c561cdb4e4a" containerID="4f7be174ecec7d1529590f28f6adacd232c18394ee05792ee3f20a86ff56216f" exitCode=7 Jul 10 19:24:37.901979 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:24:37.901570 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e61e5c8-wg2pz" event={"ID":"7dbf6ad5-f44f-4f7b-b49b-5c561cdb4e4a","Type":"ContainerDied","Data":"4f7be174ecec7d1529590f28f6adacd232c18394ee05792ee3f20a86ff56216f"} Jul 10 19:24:37.901979 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:24:37.901877 2581 scope.go:117] "RemoveContainer" containerID="4f7be174ecec7d1529590f28f6adacd232c18394ee05792ee3f20a86ff56216f" Jul 10 19:24:38.909374 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:24:38.909281 2581 generic.go:356] "Generic (PLEG): container finished" podID="7dbf6ad5-f44f-4f7b-b49b-5c561cdb4e4a" containerID="2b9079a36854ca98f2ce63469fef35f8cbf8040c4cf5b98ada42eb2990ecb103" exitCode=7 Jul 10 19:24:38.909855 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:24:38.909371 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e61e5c8-wg2pz" event={"ID":"7dbf6ad5-f44f-4f7b-b49b-5c561cdb4e4a","Type":"ContainerDied","Data":"2b9079a36854ca98f2ce63469fef35f8cbf8040c4cf5b98ada42eb2990ecb103"} Jul 10 19:24:38.909855 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:24:38.909433 2581 scope.go:117] "RemoveContainer" containerID="4f7be174ecec7d1529590f28f6adacd232c18394ee05792ee3f20a86ff56216f" Jul 10 19:24:38.909855 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:24:38.909678 2581 scope.go:117] "RemoveContainer" containerID="2b9079a36854ca98f2ce63469fef35f8cbf8040c4cf5b98ada42eb2990ecb103" Jul 10 19:24:38.910032 ip-10-0-135-67 kubenswrapper[2581]: E0710 19:24:38.910011 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6e61e5c8-wg2pz_odh-ai-gateway-infra(7dbf6ad5-f44f-4f7b-b49b-5c561cdb4e4a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e61e5c8-wg2pz" podUID="7dbf6ad5-f44f-4f7b-b49b-5c561cdb4e4a" Jul 10 19:24:39.916437 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:24:39.916405 2581 scope.go:117] "RemoveContainer" containerID="2b9079a36854ca98f2ce63469fef35f8cbf8040c4cf5b98ada42eb2990ecb103" Jul 10 19:24:39.916842 ip-10-0-135-67 kubenswrapper[2581]: E0710 19:24:39.916650 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6e61e5c8-wg2pz_odh-ai-gateway-infra(7dbf6ad5-f44f-4f7b-b49b-5c561cdb4e4a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e61e5c8-wg2pz" podUID="7dbf6ad5-f44f-4f7b-b49b-5c561cdb4e4a" Jul 10 19:24:53.536715 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:24:53.536646 2581 scope.go:117] "RemoveContainer" containerID="2b9079a36854ca98f2ce63469fef35f8cbf8040c4cf5b98ada42eb2990ecb103" Jul 10 19:24:53.995762 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:24:53.995724 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e61e5c8-wg2pz" event={"ID":"7dbf6ad5-f44f-4f7b-b49b-5c561cdb4e4a","Type":"ContainerStarted","Data":"1182364ef5249bd5fa03e26f5d822c14f29b0f3e27ee82bab72ffa0a4e90b7af"} Jul 10 19:24:55.002898 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:24:55.002860 2581 generic.go:356] "Generic (PLEG): container finished" podID="7dbf6ad5-f44f-4f7b-b49b-5c561cdb4e4a" containerID="1182364ef5249bd5fa03e26f5d822c14f29b0f3e27ee82bab72ffa0a4e90b7af" exitCode=0 Jul 10 19:24:55.003288 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:24:55.002932 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e61e5c8-wg2pz" event={"ID":"7dbf6ad5-f44f-4f7b-b49b-5c561cdb4e4a","Type":"ContainerDied","Data":"1182364ef5249bd5fa03e26f5d822c14f29b0f3e27ee82bab72ffa0a4e90b7af"} Jul 10 19:24:55.003288 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:24:55.002996 2581 scope.go:117] "RemoveContainer" containerID="2b9079a36854ca98f2ce63469fef35f8cbf8040c4cf5b98ada42eb2990ecb103" Jul 10 19:24:55.028546 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:24:55.028512 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e61e5c8-wg2pz"] Jul 10 19:24:56.154110 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:24:56.154086 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e61e5c8-wg2pz" Jul 10 19:24:56.305061 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:24:56.304961 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dbf6ad5-f44f-4f7b-b49b-5c561cdb4e4a-maas-api-service-ca\") pod \"7dbf6ad5-f44f-4f7b-b49b-5c561cdb4e4a\" (UID: \"7dbf6ad5-f44f-4f7b-b49b-5c561cdb4e4a\") " Jul 10 19:24:56.305374 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:24:56.305351 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dbf6ad5-f44f-4f7b-b49b-5c561cdb4e4a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7dbf6ad5-f44f-4f7b-b49b-5c561cdb4e4a" (UID: "7dbf6ad5-f44f-4f7b-b49b-5c561cdb4e4a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:24:56.406557 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:24:56.406507 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dbf6ad5-f44f-4f7b-b49b-5c561cdb4e4a-maas-api-service-ca\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:24:57.021819 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:24:57.021778 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e61e5c8-wg2pz" event={"ID":"7dbf6ad5-f44f-4f7b-b49b-5c561cdb4e4a","Type":"ContainerDied","Data":"f287e1ab89d39333126c05808b0706dde6f85e5657eedaf77f5305cb6d749ce6"} Jul 10 19:24:57.021819 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:24:57.021827 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f287e1ab89d39333126c05808b0706dde6f85e5657eedaf77f5305cb6d749ce6" Jul 10 19:24:57.022037 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:24:57.021795 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e61e5c8-wg2pz" Jul 10 19:24:57.062010 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:24:57.061979 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e61e5c8-wg2pz"] Jul 10 19:24:57.064245 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:24:57.064223 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e61e5c8-wg2pz"] Jul 10 19:24:58.542351 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:24:58.542318 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7dbf6ad5-f44f-4f7b-b49b-5c561cdb4e4a" path="/var/lib/kubelet/pods/7dbf6ad5-f44f-4f7b-b49b-5c561cdb4e4a/volumes" Jul 10 19:25:33.710048 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:25:33.709968 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e61e5c8-nv64n"] Jul 10 19:25:33.710494 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:25:33.710360 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dbf6ad5-f44f-4f7b-b49b-5c561cdb4e4a" containerName="revoke-keys" Jul 10 19:25:33.710494 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:25:33.710371 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dbf6ad5-f44f-4f7b-b49b-5c561cdb4e4a" containerName="revoke-keys" Jul 10 19:25:33.710494 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:25:33.710383 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dbf6ad5-f44f-4f7b-b49b-5c561cdb4e4a" containerName="revoke-keys" Jul 10 19:25:33.710494 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:25:33.710389 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dbf6ad5-f44f-4f7b-b49b-5c561cdb4e4a" containerName="revoke-keys" Jul 10 19:25:33.710494 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:25:33.710399 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dbf6ad5-f44f-4f7b-b49b-5c561cdb4e4a" containerName="revoke-keys" Jul 10 19:25:33.710494 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:25:33.710404 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dbf6ad5-f44f-4f7b-b49b-5c561cdb4e4a" containerName="revoke-keys" Jul 10 19:25:33.710494 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:25:33.710472 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dbf6ad5-f44f-4f7b-b49b-5c561cdb4e4a" containerName="revoke-keys" Jul 10 19:25:33.710494 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:25:33.710483 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dbf6ad5-f44f-4f7b-b49b-5c561cdb4e4a" containerName="revoke-keys" Jul 10 19:25:33.717408 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:25:33.717387 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e61e5c8-nv64n" Jul 10 19:25:33.720075 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:25:33.720027 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fsnnw\"" Jul 10 19:25:33.721956 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:25:33.721893 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e61e5c8-nv64n"] Jul 10 19:25:33.851039 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:25:33.851002 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9136b28-7347-4f29-9070-f367c49a40b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6e61e5c8-nv64n\" (UID: \"a9136b28-7347-4f29-9070-f367c49a40b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e61e5c8-nv64n" Jul 10 19:25:33.952134 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:25:33.952093 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9136b28-7347-4f29-9070-f367c49a40b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6e61e5c8-nv64n\" (UID: \"a9136b28-7347-4f29-9070-f367c49a40b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e61e5c8-nv64n" Jul 10 19:25:33.952794 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:25:33.952774 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9136b28-7347-4f29-9070-f367c49a40b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6e61e5c8-nv64n\" (UID: \"a9136b28-7347-4f29-9070-f367c49a40b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e61e5c8-nv64n" Jul 10 19:25:34.029038 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:25:34.029006 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e61e5c8-nv64n" Jul 10 19:25:34.165390 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:25:34.165361 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e61e5c8-nv64n"] Jul 10 19:25:34.166902 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:25:34.166872 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9136b28_7347_4f29_9070_f367c49a40b9.slice/crio-22bc41b7a1939d1eeb21ab10d7726f9d1a19a09b49a97d6ca6c3d0eea8cd4d88 WatchSource:0}: Error finding container 22bc41b7a1939d1eeb21ab10d7726f9d1a19a09b49a97d6ca6c3d0eea8cd4d88: Status 404 returned error can't find the container with id 22bc41b7a1939d1eeb21ab10d7726f9d1a19a09b49a97d6ca6c3d0eea8cd4d88 Jul 10 19:25:34.229272 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:25:34.229237 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e61e5c8-nv64n" event={"ID":"a9136b28-7347-4f29-9070-f367c49a40b9","Type":"ContainerStarted","Data":"22bc41b7a1939d1eeb21ab10d7726f9d1a19a09b49a97d6ca6c3d0eea8cd4d88"} Jul 10 19:25:35.236227 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:25:35.236138 2581 generic.go:356] "Generic (PLEG): container finished" podID="a9136b28-7347-4f29-9070-f367c49a40b9" containerID="9ec5a462716c15cd954e9e13007e699c6cccd9f9cff4d91295a22349c8f83ad7" exitCode=0 Jul 10 19:25:35.236578 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:25:35.236229 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e61e5c8-nv64n" event={"ID":"a9136b28-7347-4f29-9070-f367c49a40b9","Type":"ContainerDied","Data":"9ec5a462716c15cd954e9e13007e699c6cccd9f9cff4d91295a22349c8f83ad7"} Jul 10 19:25:36.383402 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:25:36.383379 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e61e5c8-nv64n" Jul 10 19:25:36.474039 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:25:36.474003 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9136b28-7347-4f29-9070-f367c49a40b9-maas-api-service-ca\") pod \"a9136b28-7347-4f29-9070-f367c49a40b9\" (UID: \"a9136b28-7347-4f29-9070-f367c49a40b9\") " Jul 10 19:25:36.474400 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:25:36.474376 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9136b28-7347-4f29-9070-f367c49a40b9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a9136b28-7347-4f29-9070-f367c49a40b9" (UID: "a9136b28-7347-4f29-9070-f367c49a40b9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:25:36.575223 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:25:36.575184 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9136b28-7347-4f29-9070-f367c49a40b9-maas-api-service-ca\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:25:37.248335 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:25:37.248305 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e61e5c8-nv64n" Jul 10 19:25:37.248527 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:25:37.248303 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6e61e5c8-nv64n" event={"ID":"a9136b28-7347-4f29-9070-f367c49a40b9","Type":"ContainerDied","Data":"22bc41b7a1939d1eeb21ab10d7726f9d1a19a09b49a97d6ca6c3d0eea8cd4d88"} Jul 10 19:25:37.248527 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:25:37.248466 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22bc41b7a1939d1eeb21ab10d7726f9d1a19a09b49a97d6ca6c3d0eea8cd4d88" Jul 10 19:26:05.460393 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:26:05.460357 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6402da18-qcdkl"] Jul 10 19:26:05.460902 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:26:05.460809 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9136b28-7347-4f29-9070-f367c49a40b9" containerName="revoke-keys" Jul 10 19:26:05.460902 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:26:05.460823 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9136b28-7347-4f29-9070-f367c49a40b9" containerName="revoke-keys" Jul 10 19:26:05.460982 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:26:05.460908 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dbf6ad5-f44f-4f7b-b49b-5c561cdb4e4a" containerName="revoke-keys" Jul 10 19:26:05.460982 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:26:05.460927 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9136b28-7347-4f29-9070-f367c49a40b9" containerName="revoke-keys" Jul 10 19:26:05.464209 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:26:05.464192 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6402da18-qcdkl" Jul 10 19:26:05.466779 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:26:05.466679 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fsnnw\"" Jul 10 19:26:05.472752 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:26:05.472724 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6402da18-qcdkl"] Jul 10 19:26:05.527002 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:26:05.526963 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12dfbcbd-2f3e-4fc1-99ce-a59c2c475b8d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6402da18-qcdkl\" (UID: \"12dfbcbd-2f3e-4fc1-99ce-a59c2c475b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6402da18-qcdkl" Jul 10 19:26:05.627760 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:26:05.627723 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12dfbcbd-2f3e-4fc1-99ce-a59c2c475b8d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6402da18-qcdkl\" (UID: \"12dfbcbd-2f3e-4fc1-99ce-a59c2c475b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6402da18-qcdkl" Jul 10 19:26:05.628327 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:26:05.628305 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12dfbcbd-2f3e-4fc1-99ce-a59c2c475b8d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6402da18-qcdkl\" (UID: \"12dfbcbd-2f3e-4fc1-99ce-a59c2c475b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6402da18-qcdkl" Jul 10 19:26:05.775701 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:26:05.775660 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6402da18-qcdkl" Jul 10 19:26:05.921440 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:26:05.921412 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6402da18-qcdkl"] Jul 10 19:26:05.923165 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:26:05.923140 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12dfbcbd_2f3e_4fc1_99ce_a59c2c475b8d.slice/crio-692ac72220e819f56e8cce279e36538068923f34ad3a47a804ebba897463a37d WatchSource:0}: Error finding container 692ac72220e819f56e8cce279e36538068923f34ad3a47a804ebba897463a37d: Status 404 returned error can't find the container with id 692ac72220e819f56e8cce279e36538068923f34ad3a47a804ebba897463a37d Jul 10 19:26:06.413389 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:26:06.413288 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6402da18-qcdkl" event={"ID":"12dfbcbd-2f3e-4fc1-99ce-a59c2c475b8d","Type":"ContainerStarted","Data":"773ed50e61ab83fbd3d98c6602b8ff296aedf2e6e5c5e009d68b109892e59baf"} Jul 10 19:26:06.413389 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:26:06.413348 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6402da18-qcdkl" event={"ID":"12dfbcbd-2f3e-4fc1-99ce-a59c2c475b8d","Type":"ContainerStarted","Data":"692ac72220e819f56e8cce279e36538068923f34ad3a47a804ebba897463a37d"} Jul 10 19:26:06.430125 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:26:06.430074 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6402da18-qcdkl" podStartSLOduration=1.430057702 podStartE2EDuration="1.430057702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:26:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:26:06.429651136 +0000 UTC m=+1650.473793638" watchObservedRunningTime="2026-07-10 19:26:06.430057702 +0000 UTC m=+1650.474200203" Jul 10 19:26:07.420182 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:26:07.420147 2581 generic.go:356] "Generic (PLEG): container finished" podID="12dfbcbd-2f3e-4fc1-99ce-a59c2c475b8d" containerID="773ed50e61ab83fbd3d98c6602b8ff296aedf2e6e5c5e009d68b109892e59baf" exitCode=7 Jul 10 19:26:07.420563 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:26:07.420202 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6402da18-qcdkl" event={"ID":"12dfbcbd-2f3e-4fc1-99ce-a59c2c475b8d","Type":"ContainerDied","Data":"773ed50e61ab83fbd3d98c6602b8ff296aedf2e6e5c5e009d68b109892e59baf"} Jul 10 19:26:07.420563 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:26:07.420413 2581 scope.go:117] "RemoveContainer" containerID="773ed50e61ab83fbd3d98c6602b8ff296aedf2e6e5c5e009d68b109892e59baf" Jul 10 19:26:08.427710 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:26:08.427612 2581 generic.go:356] "Generic (PLEG): container finished" podID="12dfbcbd-2f3e-4fc1-99ce-a59c2c475b8d" containerID="85df0e2c8a5075395be82ca80d20cc38c30bf0d5b851c83f5911ce28e298fb3e" exitCode=7 Jul 10 19:26:08.428140 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:26:08.427742 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6402da18-qcdkl" event={"ID":"12dfbcbd-2f3e-4fc1-99ce-a59c2c475b8d","Type":"ContainerDied","Data":"85df0e2c8a5075395be82ca80d20cc38c30bf0d5b851c83f5911ce28e298fb3e"} Jul 10 19:26:08.428140 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:26:08.427806 2581 scope.go:117] "RemoveContainer" containerID="773ed50e61ab83fbd3d98c6602b8ff296aedf2e6e5c5e009d68b109892e59baf" Jul 10 19:26:08.428140 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:26:08.427970 2581 scope.go:117] "RemoveContainer" containerID="85df0e2c8a5075395be82ca80d20cc38c30bf0d5b851c83f5911ce28e298fb3e" Jul 10 19:26:08.428294 ip-10-0-135-67 kubenswrapper[2581]: E0710 19:26:08.428206 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6402da18-qcdkl_odh-ai-gateway-infra(12dfbcbd-2f3e-4fc1-99ce-a59c2c475b8d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6402da18-qcdkl" podUID="12dfbcbd-2f3e-4fc1-99ce-a59c2c475b8d" Jul 10 19:26:09.436159 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:26:09.436131 2581 scope.go:117] "RemoveContainer" containerID="85df0e2c8a5075395be82ca80d20cc38c30bf0d5b851c83f5911ce28e298fb3e" Jul 10 19:26:09.436529 ip-10-0-135-67 kubenswrapper[2581]: E0710 19:26:09.436299 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6402da18-qcdkl_odh-ai-gateway-infra(12dfbcbd-2f3e-4fc1-99ce-a59c2c475b8d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6402da18-qcdkl" podUID="12dfbcbd-2f3e-4fc1-99ce-a59c2c475b8d" Jul 10 19:26:22.536843 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:26:22.536810 2581 scope.go:117] "RemoveContainer" containerID="85df0e2c8a5075395be82ca80d20cc38c30bf0d5b851c83f5911ce28e298fb3e" Jul 10 19:26:23.511764 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:26:23.511731 2581 generic.go:356] "Generic (PLEG): container finished" podID="12dfbcbd-2f3e-4fc1-99ce-a59c2c475b8d" containerID="4ad996bd2409fb2b910eb98cb2a76957ebfb2e5efdc7c03a879307a97b3e036d" exitCode=0 Jul 10 19:26:23.511956 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:26:23.511808 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6402da18-qcdkl" event={"ID":"12dfbcbd-2f3e-4fc1-99ce-a59c2c475b8d","Type":"ContainerDied","Data":"4ad996bd2409fb2b910eb98cb2a76957ebfb2e5efdc7c03a879307a97b3e036d"} Jul 10 19:26:23.511956 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:26:23.511891 2581 scope.go:117] "RemoveContainer" containerID="85df0e2c8a5075395be82ca80d20cc38c30bf0d5b851c83f5911ce28e298fb3e" Jul 10 19:26:24.543699 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:26:24.543653 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6402da18-qcdkl"] Jul 10 19:26:24.662331 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:26:24.662304 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6402da18-qcdkl" Jul 10 19:26:24.806525 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:26:24.806437 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12dfbcbd-2f3e-4fc1-99ce-a59c2c475b8d-maas-api-service-ca\") pod \"12dfbcbd-2f3e-4fc1-99ce-a59c2c475b8d\" (UID: \"12dfbcbd-2f3e-4fc1-99ce-a59c2c475b8d\") " Jul 10 19:26:24.806805 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:26:24.806782 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12dfbcbd-2f3e-4fc1-99ce-a59c2c475b8d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "12dfbcbd-2f3e-4fc1-99ce-a59c2c475b8d" (UID: "12dfbcbd-2f3e-4fc1-99ce-a59c2c475b8d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:26:24.907201 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:26:24.907162 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12dfbcbd-2f3e-4fc1-99ce-a59c2c475b8d-maas-api-service-ca\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:26:25.532823 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:26:25.532791 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6402da18-qcdkl" Jul 10 19:26:25.532823 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:26:25.532811 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6402da18-qcdkl" event={"ID":"12dfbcbd-2f3e-4fc1-99ce-a59c2c475b8d","Type":"ContainerDied","Data":"692ac72220e819f56e8cce279e36538068923f34ad3a47a804ebba897463a37d"} Jul 10 19:26:25.533061 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:26:25.532865 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="692ac72220e819f56e8cce279e36538068923f34ad3a47a804ebba897463a37d" Jul 10 19:26:25.565576 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:26:25.565540 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6402da18-qcdkl"] Jul 10 19:26:25.569108 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:26:25.569082 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6402da18-qcdkl"] Jul 10 19:26:26.541604 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:26:26.541570 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12dfbcbd-2f3e-4fc1-99ce-a59c2c475b8d" path="/var/lib/kubelet/pods/12dfbcbd-2f3e-4fc1-99ce-a59c2c475b8d/volumes" Jul 10 19:27:05.506796 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:05.506702 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6402da18-4vlzq"] Jul 10 19:27:05.507353 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:05.507127 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12dfbcbd-2f3e-4fc1-99ce-a59c2c475b8d" containerName="revoke-keys" Jul 10 19:27:05.507353 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:05.507140 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="12dfbcbd-2f3e-4fc1-99ce-a59c2c475b8d" containerName="revoke-keys" Jul 10 19:27:05.507353 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:05.507163 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12dfbcbd-2f3e-4fc1-99ce-a59c2c475b8d" containerName="revoke-keys" Jul 10 19:27:05.507353 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:05.507169 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="12dfbcbd-2f3e-4fc1-99ce-a59c2c475b8d" containerName="revoke-keys" Jul 10 19:27:05.507353 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:05.507178 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12dfbcbd-2f3e-4fc1-99ce-a59c2c475b8d" containerName="revoke-keys" Jul 10 19:27:05.507353 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:05.507183 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="12dfbcbd-2f3e-4fc1-99ce-a59c2c475b8d" containerName="revoke-keys" Jul 10 19:27:05.507353 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:05.507244 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="12dfbcbd-2f3e-4fc1-99ce-a59c2c475b8d" containerName="revoke-keys" Jul 10 19:27:05.507353 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:05.507251 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="12dfbcbd-2f3e-4fc1-99ce-a59c2c475b8d" containerName="revoke-keys" Jul 10 19:27:05.510377 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:05.510352 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6402da18-4vlzq" Jul 10 19:27:05.513098 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:05.513072 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fsnnw\"" Jul 10 19:27:05.519221 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:05.518959 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6402da18-4vlzq"] Jul 10 19:27:05.560207 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:05.560173 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d04de4d1-36b2-4d1e-b95b-3f5304ca217e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6402da18-4vlzq\" (UID: \"d04de4d1-36b2-4d1e-b95b-3f5304ca217e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6402da18-4vlzq" Jul 10 19:27:05.661707 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:05.661651 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d04de4d1-36b2-4d1e-b95b-3f5304ca217e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6402da18-4vlzq\" (UID: \"d04de4d1-36b2-4d1e-b95b-3f5304ca217e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6402da18-4vlzq" Jul 10 19:27:05.662340 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:05.662315 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d04de4d1-36b2-4d1e-b95b-3f5304ca217e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6402da18-4vlzq\" (UID: \"d04de4d1-36b2-4d1e-b95b-3f5304ca217e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6402da18-4vlzq" Jul 10 19:27:05.822967 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:05.822877 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6402da18-4vlzq" Jul 10 19:27:05.965258 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:05.965224 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6402da18-4vlzq"] Jul 10 19:27:05.967055 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:27:05.967020 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd04de4d1_36b2_4d1e_b95b_3f5304ca217e.slice/crio-eed882a545437fa15e6ed0d4ac72ca2206cac5f45b93a8828e484c9724c24685 WatchSource:0}: Error finding container eed882a545437fa15e6ed0d4ac72ca2206cac5f45b93a8828e484c9724c24685: Status 404 returned error can't find the container with id eed882a545437fa15e6ed0d4ac72ca2206cac5f45b93a8828e484c9724c24685 Jul 10 19:27:06.765818 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:06.765784 2581 generic.go:356] "Generic (PLEG): container finished" podID="d04de4d1-36b2-4d1e-b95b-3f5304ca217e" containerID="d80884c6a544f12e65f7f2c364ec4b95c9b45e23da312b61982fba45bcb12c0e" exitCode=0 Jul 10 19:27:06.766188 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:06.765863 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6402da18-4vlzq" event={"ID":"d04de4d1-36b2-4d1e-b95b-3f5304ca217e","Type":"ContainerDied","Data":"d80884c6a544f12e65f7f2c364ec4b95c9b45e23da312b61982fba45bcb12c0e"} Jul 10 19:27:06.766188 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:06.765918 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6402da18-4vlzq" event={"ID":"d04de4d1-36b2-4d1e-b95b-3f5304ca217e","Type":"ContainerStarted","Data":"eed882a545437fa15e6ed0d4ac72ca2206cac5f45b93a8828e484c9724c24685"} Jul 10 19:27:07.908913 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:07.908886 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6402da18-4vlzq" Jul 10 19:27:07.985004 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:07.984969 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d04de4d1-36b2-4d1e-b95b-3f5304ca217e-maas-api-service-ca\") pod \"d04de4d1-36b2-4d1e-b95b-3f5304ca217e\" (UID: \"d04de4d1-36b2-4d1e-b95b-3f5304ca217e\") " Jul 10 19:27:07.985327 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:07.985306 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d04de4d1-36b2-4d1e-b95b-3f5304ca217e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d04de4d1-36b2-4d1e-b95b-3f5304ca217e" (UID: "d04de4d1-36b2-4d1e-b95b-3f5304ca217e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:27:08.086671 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:08.086584 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d04de4d1-36b2-4d1e-b95b-3f5304ca217e-maas-api-service-ca\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:27:08.778144 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:08.778103 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6402da18-4vlzq" event={"ID":"d04de4d1-36b2-4d1e-b95b-3f5304ca217e","Type":"ContainerDied","Data":"eed882a545437fa15e6ed0d4ac72ca2206cac5f45b93a8828e484c9724c24685"} Jul 10 19:27:08.778144 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:08.778128 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6402da18-4vlzq" Jul 10 19:27:08.778144 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:08.778154 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eed882a545437fa15e6ed0d4ac72ca2206cac5f45b93a8828e484c9724c24685" Jul 10 19:27:37.384069 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:37.384035 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2397a366-kxfs4"] Jul 10 19:27:37.384502 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:37.384487 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d04de4d1-36b2-4d1e-b95b-3f5304ca217e" containerName="revoke-keys" Jul 10 19:27:37.384546 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:37.384504 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d04de4d1-36b2-4d1e-b95b-3f5304ca217e" containerName="revoke-keys" Jul 10 19:27:37.384583 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:37.384575 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d04de4d1-36b2-4d1e-b95b-3f5304ca217e" containerName="revoke-keys" Jul 10 19:27:37.384617 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:37.384585 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="12dfbcbd-2f3e-4fc1-99ce-a59c2c475b8d" containerName="revoke-keys" Jul 10 19:27:37.387931 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:37.387914 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2397a366-kxfs4" Jul 10 19:27:37.390562 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:37.390543 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fsnnw\"" Jul 10 19:27:37.396167 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:37.396140 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2397a366-kxfs4"] Jul 10 19:27:37.563735 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:37.563670 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7d7128a-c525-4926-9a33-edceddce7f9b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-2397a366-kxfs4\" (UID: \"d7d7128a-c525-4926-9a33-edceddce7f9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2397a366-kxfs4" Jul 10 19:27:37.664294 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:37.664183 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7d7128a-c525-4926-9a33-edceddce7f9b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-2397a366-kxfs4\" (UID: \"d7d7128a-c525-4926-9a33-edceddce7f9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2397a366-kxfs4" Jul 10 19:27:37.664871 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:37.664850 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7d7128a-c525-4926-9a33-edceddce7f9b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-2397a366-kxfs4\" (UID: \"d7d7128a-c525-4926-9a33-edceddce7f9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2397a366-kxfs4" Jul 10 19:27:37.700226 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:37.700193 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2397a366-kxfs4" Jul 10 19:27:37.846238 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:37.846208 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2397a366-kxfs4"] Jul 10 19:27:37.848123 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:27:37.848093 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7d7128a_c525_4926_9a33_edceddce7f9b.slice/crio-847e93490bab6b7bdc1efc0780597526587dab72dd0e7cec7c5e295b88a893b4 WatchSource:0}: Error finding container 847e93490bab6b7bdc1efc0780597526587dab72dd0e7cec7c5e295b88a893b4: Status 404 returned error can't find the container with id 847e93490bab6b7bdc1efc0780597526587dab72dd0e7cec7c5e295b88a893b4 Jul 10 19:27:37.950184 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:37.950102 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2397a366-kxfs4" event={"ID":"d7d7128a-c525-4926-9a33-edceddce7f9b","Type":"ContainerStarted","Data":"847e93490bab6b7bdc1efc0780597526587dab72dd0e7cec7c5e295b88a893b4"} Jul 10 19:27:38.957377 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:38.957279 2581 generic.go:356] "Generic (PLEG): container finished" podID="d7d7128a-c525-4926-9a33-edceddce7f9b" containerID="d59c9d0546074dda2eb90d06eb53dbc7c265bacf805d84a6e76635580dcb86fd" exitCode=7 Jul 10 19:27:38.957377 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:38.957364 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2397a366-kxfs4" event={"ID":"d7d7128a-c525-4926-9a33-edceddce7f9b","Type":"ContainerDied","Data":"d59c9d0546074dda2eb90d06eb53dbc7c265bacf805d84a6e76635580dcb86fd"} Jul 10 19:27:38.957954 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:38.957600 2581 scope.go:117] "RemoveContainer" containerID="d59c9d0546074dda2eb90d06eb53dbc7c265bacf805d84a6e76635580dcb86fd" Jul 10 19:27:39.965438 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:39.965341 2581 generic.go:356] "Generic (PLEG): container finished" podID="d7d7128a-c525-4926-9a33-edceddce7f9b" containerID="07e601856eb266305f2bc2f474ac801b893d039354a3573f8c5c647b93c6bd80" exitCode=7 Jul 10 19:27:39.965849 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:39.965432 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2397a366-kxfs4" event={"ID":"d7d7128a-c525-4926-9a33-edceddce7f9b","Type":"ContainerDied","Data":"07e601856eb266305f2bc2f474ac801b893d039354a3573f8c5c647b93c6bd80"} Jul 10 19:27:39.965849 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:39.965550 2581 scope.go:117] "RemoveContainer" containerID="d59c9d0546074dda2eb90d06eb53dbc7c265bacf805d84a6e76635580dcb86fd" Jul 10 19:27:39.965849 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:39.965607 2581 scope.go:117] "RemoveContainer" containerID="07e601856eb266305f2bc2f474ac801b893d039354a3573f8c5c647b93c6bd80" Jul 10 19:27:39.965849 ip-10-0-135-67 kubenswrapper[2581]: E0710 19:27:39.965829 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-2397a366-kxfs4_odh-ai-gateway-infra(d7d7128a-c525-4926-9a33-edceddce7f9b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2397a366-kxfs4" podUID="d7d7128a-c525-4926-9a33-edceddce7f9b" Jul 10 19:27:40.973521 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:40.973493 2581 scope.go:117] "RemoveContainer" containerID="07e601856eb266305f2bc2f474ac801b893d039354a3573f8c5c647b93c6bd80" Jul 10 19:27:40.974049 ip-10-0-135-67 kubenswrapper[2581]: E0710 19:27:40.973717 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-2397a366-kxfs4_odh-ai-gateway-infra(d7d7128a-c525-4926-9a33-edceddce7f9b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2397a366-kxfs4" podUID="d7d7128a-c525-4926-9a33-edceddce7f9b" Jul 10 19:27:52.536335 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:52.536295 2581 scope.go:117] "RemoveContainer" containerID="07e601856eb266305f2bc2f474ac801b893d039354a3573f8c5c647b93c6bd80" Jul 10 19:27:53.042999 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:53.042964 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2397a366-kxfs4" event={"ID":"d7d7128a-c525-4926-9a33-edceddce7f9b","Type":"ContainerStarted","Data":"61d8afe581e3729e758180805293ed91f8c39765247e87856a193d107244b2de"} Jul 10 19:27:53.062306 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:53.062251 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2397a366-kxfs4" podStartSLOduration=16.062235027 podStartE2EDuration="16.062235027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:27:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:27:53.060434289 +0000 UTC m=+1757.104576823" watchObservedRunningTime="2026-07-10 19:27:53.062235027 +0000 UTC m=+1757.106377526" Jul 10 19:27:54.050494 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:54.050460 2581 generic.go:356] "Generic (PLEG): container finished" podID="d7d7128a-c525-4926-9a33-edceddce7f9b" containerID="61d8afe581e3729e758180805293ed91f8c39765247e87856a193d107244b2de" exitCode=0 Jul 10 19:27:54.050929 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:54.050515 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2397a366-kxfs4" event={"ID":"d7d7128a-c525-4926-9a33-edceddce7f9b","Type":"ContainerDied","Data":"61d8afe581e3729e758180805293ed91f8c39765247e87856a193d107244b2de"} Jul 10 19:27:54.050929 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:54.050561 2581 scope.go:117] "RemoveContainer" containerID="07e601856eb266305f2bc2f474ac801b893d039354a3573f8c5c647b93c6bd80" Jul 10 19:27:54.070551 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:54.070525 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2397a366-kxfs4"] Jul 10 19:27:55.199509 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:55.199474 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2397a366-kxfs4" Jul 10 19:27:55.303422 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:55.303384 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7d7128a-c525-4926-9a33-edceddce7f9b-maas-api-service-ca\") pod \"d7d7128a-c525-4926-9a33-edceddce7f9b\" (UID: \"d7d7128a-c525-4926-9a33-edceddce7f9b\") " Jul 10 19:27:55.303758 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:55.303734 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7d7128a-c525-4926-9a33-edceddce7f9b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d7d7128a-c525-4926-9a33-edceddce7f9b" (UID: "d7d7128a-c525-4926-9a33-edceddce7f9b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:27:55.404241 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:55.404152 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7d7128a-c525-4926-9a33-edceddce7f9b-maas-api-service-ca\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:27:56.065341 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:56.065311 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2397a366-kxfs4" event={"ID":"d7d7128a-c525-4926-9a33-edceddce7f9b","Type":"ContainerDied","Data":"847e93490bab6b7bdc1efc0780597526587dab72dd0e7cec7c5e295b88a893b4"} Jul 10 19:27:56.065495 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:56.065362 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="847e93490bab6b7bdc1efc0780597526587dab72dd0e7cec7c5e295b88a893b4" Jul 10 19:27:56.065495 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:56.065323 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2397a366-kxfs4" Jul 10 19:27:56.097125 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:56.097096 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2397a366-kxfs4"] Jul 10 19:27:56.101296 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:56.101271 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2397a366-kxfs4"] Jul 10 19:27:56.549371 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:27:56.549334 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7d7128a-c525-4926-9a33-edceddce7f9b" path="/var/lib/kubelet/pods/d7d7128a-c525-4926-9a33-edceddce7f9b/volumes" Jul 10 19:28:27.418754 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:28:27.418714 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2397a366-ww45x"] Jul 10 19:28:27.419228 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:28:27.419152 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7d7128a-c525-4926-9a33-edceddce7f9b" containerName="revoke-keys" Jul 10 19:28:27.419228 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:28:27.419164 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7d7128a-c525-4926-9a33-edceddce7f9b" containerName="revoke-keys" Jul 10 19:28:27.419228 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:28:27.419177 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7d7128a-c525-4926-9a33-edceddce7f9b" containerName="revoke-keys" Jul 10 19:28:27.419228 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:28:27.419183 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7d7128a-c525-4926-9a33-edceddce7f9b" containerName="revoke-keys" Jul 10 19:28:27.419228 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:28:27.419200 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7d7128a-c525-4926-9a33-edceddce7f9b" containerName="revoke-keys" Jul 10 19:28:27.419228 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:28:27.419205 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7d7128a-c525-4926-9a33-edceddce7f9b" containerName="revoke-keys" Jul 10 19:28:27.419419 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:28:27.419277 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7d7128a-c525-4926-9a33-edceddce7f9b" containerName="revoke-keys" Jul 10 19:28:27.419419 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:28:27.419286 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7d7128a-c525-4926-9a33-edceddce7f9b" containerName="revoke-keys" Jul 10 19:28:27.422356 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:28:27.422339 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2397a366-ww45x" Jul 10 19:28:27.424638 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:28:27.424615 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fsnnw\"" Jul 10 19:28:27.429945 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:28:27.429919 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2397a366-ww45x"] Jul 10 19:28:27.497537 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:28:27.497498 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad749dff-107e-4587-a8f8-4199fd77971e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-2397a366-ww45x\" (UID: \"ad749dff-107e-4587-a8f8-4199fd77971e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2397a366-ww45x" Jul 10 19:28:27.598265 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:28:27.598224 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad749dff-107e-4587-a8f8-4199fd77971e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-2397a366-ww45x\" (UID: \"ad749dff-107e-4587-a8f8-4199fd77971e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2397a366-ww45x" Jul 10 19:28:27.598859 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:28:27.598836 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad749dff-107e-4587-a8f8-4199fd77971e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-2397a366-ww45x\" (UID: \"ad749dff-107e-4587-a8f8-4199fd77971e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2397a366-ww45x" Jul 10 19:28:27.735362 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:28:27.735268 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2397a366-ww45x" Jul 10 19:28:27.877030 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:28:27.877005 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2397a366-ww45x"] Jul 10 19:28:27.881650 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:28:27.881615 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad749dff_107e_4587_a8f8_4199fd77971e.slice/crio-173bcc5ef4eec0a6c57250bab3cf130c49fab51e1f41c0444c5437f701e80995 WatchSource:0}: Error finding container 173bcc5ef4eec0a6c57250bab3cf130c49fab51e1f41c0444c5437f701e80995: Status 404 returned error can't find the container with id 173bcc5ef4eec0a6c57250bab3cf130c49fab51e1f41c0444c5437f701e80995 Jul 10 19:28:28.244667 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:28:28.244622 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2397a366-ww45x" event={"ID":"ad749dff-107e-4587-a8f8-4199fd77971e","Type":"ContainerStarted","Data":"b0bf2f4284b4781a6e9cf4369144a747e829f903a13b13eb289f124b940a9c28"} Jul 10 19:28:28.244890 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:28:28.244705 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2397a366-ww45x" event={"ID":"ad749dff-107e-4587-a8f8-4199fd77971e","Type":"ContainerStarted","Data":"173bcc5ef4eec0a6c57250bab3cf130c49fab51e1f41c0444c5437f701e80995"} Jul 10 19:28:28.262954 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:28:28.262900 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2397a366-ww45x" podStartSLOduration=1.2628824220000001 podStartE2EDuration="1.262882422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:28:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:28:28.260855159 +0000 UTC m=+1792.304997661" watchObservedRunningTime="2026-07-10 19:28:28.262882422 +0000 UTC m=+1792.307024923" Jul 10 19:28:29.251507 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:28:29.251458 2581 generic.go:356] "Generic (PLEG): container finished" podID="ad749dff-107e-4587-a8f8-4199fd77971e" containerID="b0bf2f4284b4781a6e9cf4369144a747e829f903a13b13eb289f124b940a9c28" exitCode=0 Jul 10 19:28:29.251895 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:28:29.251515 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2397a366-ww45x" event={"ID":"ad749dff-107e-4587-a8f8-4199fd77971e","Type":"ContainerDied","Data":"b0bf2f4284b4781a6e9cf4369144a747e829f903a13b13eb289f124b940a9c28"} Jul 10 19:28:30.394251 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:28:30.394226 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2397a366-ww45x" Jul 10 19:28:30.527462 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:28:30.527375 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad749dff-107e-4587-a8f8-4199fd77971e-maas-api-service-ca\") pod \"ad749dff-107e-4587-a8f8-4199fd77971e\" (UID: \"ad749dff-107e-4587-a8f8-4199fd77971e\") " Jul 10 19:28:30.527758 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:28:30.527732 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad749dff-107e-4587-a8f8-4199fd77971e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ad749dff-107e-4587-a8f8-4199fd77971e" (UID: "ad749dff-107e-4587-a8f8-4199fd77971e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:28:30.628837 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:28:30.628796 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad749dff-107e-4587-a8f8-4199fd77971e-maas-api-service-ca\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:28:31.264232 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:28:31.264200 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2397a366-ww45x" Jul 10 19:28:31.264434 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:28:31.264194 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2397a366-ww45x" event={"ID":"ad749dff-107e-4587-a8f8-4199fd77971e","Type":"ContainerDied","Data":"173bcc5ef4eec0a6c57250bab3cf130c49fab51e1f41c0444c5437f701e80995"} Jul 10 19:28:31.264434 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:28:31.264355 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="173bcc5ef4eec0a6c57250bab3cf130c49fab51e1f41c0444c5437f701e80995" Jul 10 19:28:36.633772 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:28:36.633744 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69hnr_7f479cec-2fb8-45c4-ae7d-073194a5dede/ovn-acl-logging/0.log" Jul 10 19:28:36.644727 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:28:36.644702 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69hnr_7f479cec-2fb8-45c4-ae7d-073194a5dede/ovn-acl-logging/0.log" Jul 10 19:30:44.415638 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:30:44.415590 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d2414db-q2clr"] Jul 10 19:30:44.416200 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:30:44.416019 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad749dff-107e-4587-a8f8-4199fd77971e" containerName="revoke-keys" Jul 10 19:30:44.416200 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:30:44.416032 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad749dff-107e-4587-a8f8-4199fd77971e" containerName="revoke-keys" Jul 10 19:30:44.416200 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:30:44.416112 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad749dff-107e-4587-a8f8-4199fd77971e" containerName="revoke-keys" Jul 10 19:30:44.416200 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:30:44.416119 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7d7128a-c525-4926-9a33-edceddce7f9b" containerName="revoke-keys" Jul 10 19:30:44.419105 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:30:44.419082 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d2414db-q2clr" Jul 10 19:30:44.421399 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:30:44.421376 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fsnnw\"" Jul 10 19:30:44.426254 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:30:44.426230 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d2414db-q2clr"] Jul 10 19:30:44.531056 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:30:44.531020 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dca83456-8418-41ac-8205-bdfb27251795-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7d2414db-q2clr\" (UID: \"dca83456-8418-41ac-8205-bdfb27251795\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d2414db-q2clr" Jul 10 19:30:44.631879 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:30:44.631835 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dca83456-8418-41ac-8205-bdfb27251795-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7d2414db-q2clr\" (UID: \"dca83456-8418-41ac-8205-bdfb27251795\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d2414db-q2clr" Jul 10 19:30:44.632527 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:30:44.632499 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dca83456-8418-41ac-8205-bdfb27251795-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7d2414db-q2clr\" (UID: \"dca83456-8418-41ac-8205-bdfb27251795\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d2414db-q2clr" Jul 10 19:30:44.732330 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:30:44.732244 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d2414db-q2clr" Jul 10 19:30:44.875227 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:30:44.875191 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d2414db-q2clr"] Jul 10 19:30:44.875475 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:30:44.875448 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddca83456_8418_41ac_8205_bdfb27251795.slice/crio-b222ace5dcebdcc09681c7f63cae28d8f8b44ca95fac1cb75d170e1a123c64e5 WatchSource:0}: Error finding container b222ace5dcebdcc09681c7f63cae28d8f8b44ca95fac1cb75d170e1a123c64e5: Status 404 returned error can't find the container with id b222ace5dcebdcc09681c7f63cae28d8f8b44ca95fac1cb75d170e1a123c64e5 Jul 10 19:30:45.022719 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:30:45.022673 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d2414db-q2clr" event={"ID":"dca83456-8418-41ac-8205-bdfb27251795","Type":"ContainerStarted","Data":"b222ace5dcebdcc09681c7f63cae28d8f8b44ca95fac1cb75d170e1a123c64e5"} Jul 10 19:30:46.030384 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:30:46.030345 2581 generic.go:356] "Generic (PLEG): container finished" podID="dca83456-8418-41ac-8205-bdfb27251795" containerID="8cb80e385fd150780ee6c83a08d58c5d6badb01f787500700564480740128068" exitCode=0 Jul 10 19:30:46.030815 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:30:46.030434 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d2414db-q2clr" event={"ID":"dca83456-8418-41ac-8205-bdfb27251795","Type":"ContainerDied","Data":"8cb80e385fd150780ee6c83a08d58c5d6badb01f787500700564480740128068"} Jul 10 19:30:47.173834 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:30:47.173807 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d2414db-q2clr" Jul 10 19:30:47.257301 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:30:47.257268 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dca83456-8418-41ac-8205-bdfb27251795-maas-api-service-ca\") pod \"dca83456-8418-41ac-8205-bdfb27251795\" (UID: \"dca83456-8418-41ac-8205-bdfb27251795\") " Jul 10 19:30:47.257610 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:30:47.257589 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dca83456-8418-41ac-8205-bdfb27251795-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dca83456-8418-41ac-8205-bdfb27251795" (UID: "dca83456-8418-41ac-8205-bdfb27251795"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:30:47.358923 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:30:47.358841 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dca83456-8418-41ac-8205-bdfb27251795-maas-api-service-ca\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:30:48.042070 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:30:48.042045 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d2414db-q2clr" Jul 10 19:30:48.042249 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:30:48.042045 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7d2414db-q2clr" event={"ID":"dca83456-8418-41ac-8205-bdfb27251795","Type":"ContainerDied","Data":"b222ace5dcebdcc09681c7f63cae28d8f8b44ca95fac1cb75d170e1a123c64e5"} Jul 10 19:30:48.042249 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:30:48.042162 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b222ace5dcebdcc09681c7f63cae28d8f8b44ca95fac1cb75d170e1a123c64e5" Jul 10 19:31:36.723603 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:31:36.723519 2581 scope.go:117] "RemoveContainer" containerID="1182364ef5249bd5fa03e26f5d822c14f29b0f3e27ee82bab72ffa0a4e90b7af" Jul 10 19:32:25.737446 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:32:25.737397 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cacc33b0-frmhx"] Jul 10 19:32:25.738032 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:32:25.737839 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dca83456-8418-41ac-8205-bdfb27251795" containerName="revoke-keys" Jul 10 19:32:25.738032 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:32:25.737856 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="dca83456-8418-41ac-8205-bdfb27251795" containerName="revoke-keys" Jul 10 19:32:25.738032 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:32:25.737964 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="dca83456-8418-41ac-8205-bdfb27251795" containerName="revoke-keys" Jul 10 19:32:25.740970 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:32:25.740952 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cacc33b0-frmhx" Jul 10 19:32:25.743236 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:32:25.743219 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fsnnw\"" Jul 10 19:32:25.748531 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:32:25.748503 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cacc33b0-frmhx"] Jul 10 19:32:25.930327 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:32:25.930294 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d366571-e21b-48d5-b355-7085715b7e8f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-cacc33b0-frmhx\" (UID: \"3d366571-e21b-48d5-b355-7085715b7e8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cacc33b0-frmhx" Jul 10 19:32:26.031810 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:32:26.031768 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d366571-e21b-48d5-b355-7085715b7e8f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-cacc33b0-frmhx\" (UID: \"3d366571-e21b-48d5-b355-7085715b7e8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cacc33b0-frmhx" Jul 10 19:32:26.032476 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:32:26.032455 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d366571-e21b-48d5-b355-7085715b7e8f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-cacc33b0-frmhx\" (UID: \"3d366571-e21b-48d5-b355-7085715b7e8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cacc33b0-frmhx" Jul 10 19:32:26.053707 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:32:26.053661 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cacc33b0-frmhx" Jul 10 19:32:26.195757 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:32:26.195727 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cacc33b0-frmhx"] Jul 10 19:32:26.196860 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:32:26.196834 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d366571_e21b_48d5_b355_7085715b7e8f.slice/crio-d4d1c793a6af016bff20b71a6b06f99c63b8fbdad2f7ff6c8b826c4e28f20812 WatchSource:0}: Error finding container d4d1c793a6af016bff20b71a6b06f99c63b8fbdad2f7ff6c8b826c4e28f20812: Status 404 returned error can't find the container with id d4d1c793a6af016bff20b71a6b06f99c63b8fbdad2f7ff6c8b826c4e28f20812 Jul 10 19:32:26.615466 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:32:26.615429 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cacc33b0-frmhx" event={"ID":"3d366571-e21b-48d5-b355-7085715b7e8f","Type":"ContainerStarted","Data":"52ae6467e44c60d89202bed2bdf392c9e213cab261c55929cb5a5b5d1d8180bd"} Jul 10 19:32:26.615650 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:32:26.615491 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cacc33b0-frmhx" event={"ID":"3d366571-e21b-48d5-b355-7085715b7e8f","Type":"ContainerStarted","Data":"d4d1c793a6af016bff20b71a6b06f99c63b8fbdad2f7ff6c8b826c4e28f20812"} Jul 10 19:32:26.632915 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:32:26.632865 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cacc33b0-frmhx" podStartSLOduration=1.632849719 podStartE2EDuration="1.632849719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:32:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:32:26.631426922 +0000 UTC m=+2030.675569411" watchObservedRunningTime="2026-07-10 19:32:26.632849719 +0000 UTC m=+2030.676992220" Jul 10 19:32:27.623566 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:32:27.623526 2581 generic.go:356] "Generic (PLEG): container finished" podID="3d366571-e21b-48d5-b355-7085715b7e8f" containerID="52ae6467e44c60d89202bed2bdf392c9e213cab261c55929cb5a5b5d1d8180bd" exitCode=0 Jul 10 19:32:27.624001 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:32:27.623614 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cacc33b0-frmhx" event={"ID":"3d366571-e21b-48d5-b355-7085715b7e8f","Type":"ContainerDied","Data":"52ae6467e44c60d89202bed2bdf392c9e213cab261c55929cb5a5b5d1d8180bd"} Jul 10 19:32:28.770978 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:32:28.770948 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cacc33b0-frmhx" Jul 10 19:32:28.855171 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:32:28.855138 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d366571-e21b-48d5-b355-7085715b7e8f-maas-api-service-ca\") pod \"3d366571-e21b-48d5-b355-7085715b7e8f\" (UID: \"3d366571-e21b-48d5-b355-7085715b7e8f\") " Jul 10 19:32:28.855476 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:32:28.855454 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d366571-e21b-48d5-b355-7085715b7e8f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3d366571-e21b-48d5-b355-7085715b7e8f" (UID: "3d366571-e21b-48d5-b355-7085715b7e8f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:32:28.956555 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:32:28.956460 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d366571-e21b-48d5-b355-7085715b7e8f-maas-api-service-ca\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:32:29.638922 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:32:29.638878 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cacc33b0-frmhx" event={"ID":"3d366571-e21b-48d5-b355-7085715b7e8f","Type":"ContainerDied","Data":"d4d1c793a6af016bff20b71a6b06f99c63b8fbdad2f7ff6c8b826c4e28f20812"} Jul 10 19:32:29.638922 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:32:29.638918 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cacc33b0-frmhx" Jul 10 19:32:29.639132 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:32:29.638934 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4d1c793a6af016bff20b71a6b06f99c63b8fbdad2f7ff6c8b826c4e28f20812" Jul 10 19:32:36.747732 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:32:36.747700 2581 scope.go:117] "RemoveContainer" containerID="4ad996bd2409fb2b910eb98cb2a76957ebfb2e5efdc7c03a879307a97b3e036d" Jul 10 19:33:36.671573 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:33:36.671540 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69hnr_7f479cec-2fb8-45c4-ae7d-073194a5dede/ovn-acl-logging/0.log" Jul 10 19:33:36.682840 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:33:36.682813 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69hnr_7f479cec-2fb8-45c4-ae7d-073194a5dede/ovn-acl-logging/0.log" Jul 10 19:34:36.778114 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:34:36.778030 2581 scope.go:117] "RemoveContainer" containerID="61d8afe581e3729e758180805293ed91f8c39765247e87856a193d107244b2de" Jul 10 19:36:20.497345 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:36:20.497261 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd22f1-v7cck"] Jul 10 19:36:20.497858 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:36:20.497674 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d366571-e21b-48d5-b355-7085715b7e8f" containerName="revoke-keys" Jul 10 19:36:20.497858 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:36:20.497698 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d366571-e21b-48d5-b355-7085715b7e8f" containerName="revoke-keys" Jul 10 19:36:20.497858 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:36:20.497768 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d366571-e21b-48d5-b355-7085715b7e8f" containerName="revoke-keys" Jul 10 19:36:20.500968 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:36:20.500933 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd22f1-v7cck" Jul 10 19:36:20.503467 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:36:20.503444 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fsnnw\"" Jul 10 19:36:20.508432 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:36:20.508403 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd22f1-v7cck"] Jul 10 19:36:20.585774 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:36:20.585722 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0317525e-4866-4139-a490-1a540ab90893-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fd22f1-v7cck\" (UID: \"0317525e-4866-4139-a490-1a540ab90893\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd22f1-v7cck" Jul 10 19:36:20.686807 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:36:20.686771 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0317525e-4866-4139-a490-1a540ab90893-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fd22f1-v7cck\" (UID: \"0317525e-4866-4139-a490-1a540ab90893\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd22f1-v7cck" Jul 10 19:36:20.687397 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:36:20.687377 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0317525e-4866-4139-a490-1a540ab90893-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fd22f1-v7cck\" (UID: \"0317525e-4866-4139-a490-1a540ab90893\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd22f1-v7cck" Jul 10 19:36:20.813982 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:36:20.813937 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd22f1-v7cck" Jul 10 19:36:20.958643 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:36:20.958192 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd22f1-v7cck"] Jul 10 19:36:20.958643 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:36:20.958212 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0317525e_4866_4139_a490_1a540ab90893.slice/crio-446fb496deac36b9e9e1e6bc480ec78b541e58427ccfd983f5ccb4238d7341aa WatchSource:0}: Error finding container 446fb496deac36b9e9e1e6bc480ec78b541e58427ccfd983f5ccb4238d7341aa: Status 404 returned error can't find the container with id 446fb496deac36b9e9e1e6bc480ec78b541e58427ccfd983f5ccb4238d7341aa Jul 10 19:36:20.977578 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:36:20.977549 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd22f1-v7cck" event={"ID":"0317525e-4866-4139-a490-1a540ab90893","Type":"ContainerStarted","Data":"446fb496deac36b9e9e1e6bc480ec78b541e58427ccfd983f5ccb4238d7341aa"} Jul 10 19:36:21.984075 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:36:21.984037 2581 generic.go:356] "Generic (PLEG): container finished" podID="0317525e-4866-4139-a490-1a540ab90893" containerID="ea770567e9a6bd4bb0c0a2b03bebd4eb9d01fb47191834bcca53c3380dbfb18f" exitCode=0 Jul 10 19:36:21.984462 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:36:21.984085 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd22f1-v7cck" event={"ID":"0317525e-4866-4139-a490-1a540ab90893","Type":"ContainerDied","Data":"ea770567e9a6bd4bb0c0a2b03bebd4eb9d01fb47191834bcca53c3380dbfb18f"} Jul 10 19:36:23.146879 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:36:23.146546 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd22f1-v7cck" Jul 10 19:36:23.203091 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:36:23.203058 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0317525e-4866-4139-a490-1a540ab90893-maas-api-service-ca\") pod \"0317525e-4866-4139-a490-1a540ab90893\" (UID: \"0317525e-4866-4139-a490-1a540ab90893\") " Jul 10 19:36:23.203403 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:36:23.203381 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0317525e-4866-4139-a490-1a540ab90893-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0317525e-4866-4139-a490-1a540ab90893" (UID: "0317525e-4866-4139-a490-1a540ab90893"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:36:23.303975 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:36:23.303939 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0317525e-4866-4139-a490-1a540ab90893-maas-api-service-ca\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:36:23.997980 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:36:23.997944 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd22f1-v7cck" event={"ID":"0317525e-4866-4139-a490-1a540ab90893","Type":"ContainerDied","Data":"446fb496deac36b9e9e1e6bc480ec78b541e58427ccfd983f5ccb4238d7341aa"} Jul 10 19:36:23.997980 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:36:23.997968 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fd22f1-v7cck" Jul 10 19:36:23.998199 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:36:23.997997 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="446fb496deac36b9e9e1e6bc480ec78b541e58427ccfd983f5ccb4238d7341aa" Jul 10 19:36:46.852899 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:36:46.852862 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8c42c2-5smpw"] Jul 10 19:36:46.853313 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:36:46.853288 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0317525e-4866-4139-a490-1a540ab90893" containerName="revoke-keys" Jul 10 19:36:46.853313 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:36:46.853299 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0317525e-4866-4139-a490-1a540ab90893" containerName="revoke-keys" Jul 10 19:36:46.853386 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:36:46.853358 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0317525e-4866-4139-a490-1a540ab90893" containerName="revoke-keys" Jul 10 19:36:46.856628 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:36:46.856610 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8c42c2-5smpw" Jul 10 19:36:46.859240 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:36:46.859220 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fsnnw\"" Jul 10 19:36:46.864599 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:36:46.864571 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8c42c2-5smpw"] Jul 10 19:36:46.902229 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:36:46.902181 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2bfe8ce-f4a4-4620-b3a1-ac3a2e0ecafa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8c42c2-5smpw\" (UID: \"c2bfe8ce-f4a4-4620-b3a1-ac3a2e0ecafa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8c42c2-5smpw" Jul 10 19:36:47.002891 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:36:47.002852 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2bfe8ce-f4a4-4620-b3a1-ac3a2e0ecafa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8c42c2-5smpw\" (UID: \"c2bfe8ce-f4a4-4620-b3a1-ac3a2e0ecafa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8c42c2-5smpw" Jul 10 19:36:47.003549 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:36:47.003525 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2bfe8ce-f4a4-4620-b3a1-ac3a2e0ecafa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8c42c2-5smpw\" (UID: \"c2bfe8ce-f4a4-4620-b3a1-ac3a2e0ecafa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8c42c2-5smpw" Jul 10 19:36:47.169392 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:36:47.169301 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8c42c2-5smpw" Jul 10 19:36:47.315420 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:36:47.315389 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8c42c2-5smpw"] Jul 10 19:36:47.317155 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:36:47.317129 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2bfe8ce_f4a4_4620_b3a1_ac3a2e0ecafa.slice/crio-424b54aaa6edaeee9c1e6877d30315acee5bc216eec2042f5e68e5be639faec0 WatchSource:0}: Error finding container 424b54aaa6edaeee9c1e6877d30315acee5bc216eec2042f5e68e5be639faec0: Status 404 returned error can't find the container with id 424b54aaa6edaeee9c1e6877d30315acee5bc216eec2042f5e68e5be639faec0 Jul 10 19:36:48.148297 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:36:48.148271 2581 generic.go:356] "Generic (PLEG): container finished" podID="c2bfe8ce-f4a4-4620-b3a1-ac3a2e0ecafa" containerID="c1e358e95fec98a2c110794b45d24c27e5815f828b87cbdf9bfbaa1c87de3ec8" exitCode=0 Jul 10 19:36:48.148611 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:36:48.148348 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8c42c2-5smpw" event={"ID":"c2bfe8ce-f4a4-4620-b3a1-ac3a2e0ecafa","Type":"ContainerDied","Data":"c1e358e95fec98a2c110794b45d24c27e5815f828b87cbdf9bfbaa1c87de3ec8"} Jul 10 19:36:48.148611 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:36:48.148400 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8c42c2-5smpw" event={"ID":"c2bfe8ce-f4a4-4620-b3a1-ac3a2e0ecafa","Type":"ContainerStarted","Data":"424b54aaa6edaeee9c1e6877d30315acee5bc216eec2042f5e68e5be639faec0"} Jul 10 19:36:49.291363 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:36:49.291334 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8c42c2-5smpw" Jul 10 19:36:49.426146 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:36:49.426044 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2bfe8ce-f4a4-4620-b3a1-ac3a2e0ecafa-maas-api-service-ca\") pod \"c2bfe8ce-f4a4-4620-b3a1-ac3a2e0ecafa\" (UID: \"c2bfe8ce-f4a4-4620-b3a1-ac3a2e0ecafa\") " Jul 10 19:36:49.426426 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:36:49.426403 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2bfe8ce-f4a4-4620-b3a1-ac3a2e0ecafa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c2bfe8ce-f4a4-4620-b3a1-ac3a2e0ecafa" (UID: "c2bfe8ce-f4a4-4620-b3a1-ac3a2e0ecafa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:36:49.527613 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:36:49.527577 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2bfe8ce-f4a4-4620-b3a1-ac3a2e0ecafa-maas-api-service-ca\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:36:50.161325 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:36:50.161291 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8c42c2-5smpw" event={"ID":"c2bfe8ce-f4a4-4620-b3a1-ac3a2e0ecafa","Type":"ContainerDied","Data":"424b54aaa6edaeee9c1e6877d30315acee5bc216eec2042f5e68e5be639faec0"} Jul 10 19:36:50.161504 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:36:50.161346 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="424b54aaa6edaeee9c1e6877d30315acee5bc216eec2042f5e68e5be639faec0" Jul 10 19:36:50.161504 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:36:50.161300 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8c42c2-5smpw" Jul 10 19:37:37.773141 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:37:37.773101 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9d1921-wd9fx"] Jul 10 19:37:37.773802 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:37:37.773781 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2bfe8ce-f4a4-4620-b3a1-ac3a2e0ecafa" containerName="revoke-keys" Jul 10 19:37:37.773871 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:37:37.773806 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2bfe8ce-f4a4-4620-b3a1-ac3a2e0ecafa" containerName="revoke-keys" Jul 10 19:37:37.773940 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:37:37.773928 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2bfe8ce-f4a4-4620-b3a1-ac3a2e0ecafa" containerName="revoke-keys" Jul 10 19:37:37.777894 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:37:37.777876 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9d1921-wd9fx" Jul 10 19:37:37.780382 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:37:37.780354 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fsnnw\"" Jul 10 19:37:37.783159 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:37:37.783136 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9d1921-wd9fx"] Jul 10 19:37:37.848658 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:37:37.848625 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbe338d5-3c79-4bc2-a33b-07a7d5037493-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9d1921-wd9fx\" (UID: \"fbe338d5-3c79-4bc2-a33b-07a7d5037493\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9d1921-wd9fx" Jul 10 19:37:37.950087 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:37:37.950051 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbe338d5-3c79-4bc2-a33b-07a7d5037493-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9d1921-wd9fx\" (UID: \"fbe338d5-3c79-4bc2-a33b-07a7d5037493\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9d1921-wd9fx" Jul 10 19:37:37.950759 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:37:37.950738 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbe338d5-3c79-4bc2-a33b-07a7d5037493-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9d1921-wd9fx\" (UID: \"fbe338d5-3c79-4bc2-a33b-07a7d5037493\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9d1921-wd9fx" Jul 10 19:37:38.090780 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:37:38.090676 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9d1921-wd9fx" Jul 10 19:37:38.228460 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:37:38.228429 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9d1921-wd9fx"] Jul 10 19:37:38.229556 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:37:38.229531 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbe338d5_3c79_4bc2_a33b_07a7d5037493.slice/crio-96c3776974b4c2a5d5bb35160dd052e43d3e712af4ee8827f0a266e6690c2b0e WatchSource:0}: Error finding container 96c3776974b4c2a5d5bb35160dd052e43d3e712af4ee8827f0a266e6690c2b0e: Status 404 returned error can't find the container with id 96c3776974b4c2a5d5bb35160dd052e43d3e712af4ee8827f0a266e6690c2b0e Jul 10 19:37:38.457843 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:37:38.457814 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9d1921-wd9fx" event={"ID":"fbe338d5-3c79-4bc2-a33b-07a7d5037493","Type":"ContainerStarted","Data":"96c3776974b4c2a5d5bb35160dd052e43d3e712af4ee8827f0a266e6690c2b0e"} Jul 10 19:37:39.471790 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:37:39.471753 2581 generic.go:356] "Generic (PLEG): container finished" podID="fbe338d5-3c79-4bc2-a33b-07a7d5037493" containerID="f4d03d92c883f3444f09b54cdcd6f3c91a7cae91a4b90e241eeb385aa8f3576e" exitCode=0 Jul 10 19:37:39.472192 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:37:39.471853 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9d1921-wd9fx" event={"ID":"fbe338d5-3c79-4bc2-a33b-07a7d5037493","Type":"ContainerDied","Data":"f4d03d92c883f3444f09b54cdcd6f3c91a7cae91a4b90e241eeb385aa8f3576e"} Jul 10 19:37:40.619041 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:37:40.619015 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9d1921-wd9fx" Jul 10 19:37:40.678105 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:37:40.678071 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbe338d5-3c79-4bc2-a33b-07a7d5037493-maas-api-service-ca\") pod \"fbe338d5-3c79-4bc2-a33b-07a7d5037493\" (UID: \"fbe338d5-3c79-4bc2-a33b-07a7d5037493\") " Jul 10 19:37:40.678430 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:37:40.678405 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbe338d5-3c79-4bc2-a33b-07a7d5037493-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fbe338d5-3c79-4bc2-a33b-07a7d5037493" (UID: "fbe338d5-3c79-4bc2-a33b-07a7d5037493"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:37:40.779420 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:37:40.779386 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbe338d5-3c79-4bc2-a33b-07a7d5037493-maas-api-service-ca\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:37:41.485006 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:37:41.484977 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9d1921-wd9fx" Jul 10 19:37:41.485184 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:37:41.485007 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9d1921-wd9fx" event={"ID":"fbe338d5-3c79-4bc2-a33b-07a7d5037493","Type":"ContainerDied","Data":"96c3776974b4c2a5d5bb35160dd052e43d3e712af4ee8827f0a266e6690c2b0e"} Jul 10 19:37:41.485184 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:37:41.485061 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96c3776974b4c2a5d5bb35160dd052e43d3e712af4ee8827f0a266e6690c2b0e" Jul 10 19:38:04.142017 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:38:04.141978 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e78db6-r4gfj"] Jul 10 19:38:04.142544 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:38:04.142471 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbe338d5-3c79-4bc2-a33b-07a7d5037493" containerName="revoke-keys" Jul 10 19:38:04.142544 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:38:04.142490 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbe338d5-3c79-4bc2-a33b-07a7d5037493" containerName="revoke-keys" Jul 10 19:38:04.142648 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:38:04.142606 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbe338d5-3c79-4bc2-a33b-07a7d5037493" containerName="revoke-keys" Jul 10 19:38:04.146062 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:38:04.146042 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e78db6-r4gfj" Jul 10 19:38:04.148726 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:38:04.148672 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fsnnw\"" Jul 10 19:38:04.154395 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:38:04.154368 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e78db6-r4gfj"] Jul 10 19:38:04.297518 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:38:04.297474 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d956e2b-c623-450b-88b7-9c0bf1beb105-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e78db6-r4gfj\" (UID: \"7d956e2b-c623-450b-88b7-9c0bf1beb105\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e78db6-r4gfj" Jul 10 19:38:04.398728 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:38:04.398619 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d956e2b-c623-450b-88b7-9c0bf1beb105-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e78db6-r4gfj\" (UID: \"7d956e2b-c623-450b-88b7-9c0bf1beb105\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e78db6-r4gfj" Jul 10 19:38:04.399282 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:38:04.399258 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d956e2b-c623-450b-88b7-9c0bf1beb105-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e78db6-r4gfj\" (UID: \"7d956e2b-c623-450b-88b7-9c0bf1beb105\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e78db6-r4gfj" Jul 10 19:38:04.459235 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:38:04.459198 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e78db6-r4gfj" Jul 10 19:38:04.598598 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:38:04.598571 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e78db6-r4gfj"] Jul 10 19:38:04.600665 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:38:04.600637 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d956e2b_c623_450b_88b7_9c0bf1beb105.slice/crio-c10a299ee4d6d7aebd4dfcedef1823dcffdc323af3172615bd09969cdf978e05 WatchSource:0}: Error finding container c10a299ee4d6d7aebd4dfcedef1823dcffdc323af3172615bd09969cdf978e05: Status 404 returned error can't find the container with id c10a299ee4d6d7aebd4dfcedef1823dcffdc323af3172615bd09969cdf978e05 Jul 10 19:38:04.619956 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:38:04.619933 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e78db6-r4gfj" event={"ID":"7d956e2b-c623-450b-88b7-9c0bf1beb105","Type":"ContainerStarted","Data":"c10a299ee4d6d7aebd4dfcedef1823dcffdc323af3172615bd09969cdf978e05"} Jul 10 19:38:05.627558 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:38:05.627450 2581 generic.go:356] "Generic (PLEG): container finished" podID="7d956e2b-c623-450b-88b7-9c0bf1beb105" containerID="474b93b96c6a08499d9af81791f3bb3f6736b57f8d47bf3cebf1ab35f77650ed" exitCode=0 Jul 10 19:38:05.627972 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:38:05.627544 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e78db6-r4gfj" event={"ID":"7d956e2b-c623-450b-88b7-9c0bf1beb105","Type":"ContainerDied","Data":"474b93b96c6a08499d9af81791f3bb3f6736b57f8d47bf3cebf1ab35f77650ed"} Jul 10 19:38:06.776380 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:38:06.776356 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e78db6-r4gfj" Jul 10 19:38:06.923233 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:38:06.923144 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d956e2b-c623-450b-88b7-9c0bf1beb105-maas-api-service-ca\") pod \"7d956e2b-c623-450b-88b7-9c0bf1beb105\" (UID: \"7d956e2b-c623-450b-88b7-9c0bf1beb105\") " Jul 10 19:38:06.923531 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:38:06.923504 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d956e2b-c623-450b-88b7-9c0bf1beb105-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7d956e2b-c623-450b-88b7-9c0bf1beb105" (UID: "7d956e2b-c623-450b-88b7-9c0bf1beb105"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:38:07.024131 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:38:07.024091 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d956e2b-c623-450b-88b7-9c0bf1beb105-maas-api-service-ca\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:38:07.640797 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:38:07.640765 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e78db6-r4gfj" Jul 10 19:38:07.640797 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:38:07.640779 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e78db6-r4gfj" event={"ID":"7d956e2b-c623-450b-88b7-9c0bf1beb105","Type":"ContainerDied","Data":"c10a299ee4d6d7aebd4dfcedef1823dcffdc323af3172615bd09969cdf978e05"} Jul 10 19:38:07.641036 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:38:07.640829 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c10a299ee4d6d7aebd4dfcedef1823dcffdc323af3172615bd09969cdf978e05" Jul 10 19:38:36.710997 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:38:36.710968 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69hnr_7f479cec-2fb8-45c4-ae7d-073194a5dede/ovn-acl-logging/0.log" Jul 10 19:38:36.724259 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:38:36.724228 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69hnr_7f479cec-2fb8-45c4-ae7d-073194a5dede/ovn-acl-logging/0.log" Jul 10 19:43:36.749805 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:43:36.749728 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69hnr_7f479cec-2fb8-45c4-ae7d-073194a5dede/ovn-acl-logging/0.log" Jul 10 19:43:36.764534 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:43:36.764490 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69hnr_7f479cec-2fb8-45c4-ae7d-073194a5dede/ovn-acl-logging/0.log" Jul 10 19:44:14.863464 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:14.863423 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7346-139701951232880"] Jul 10 19:44:14.864017 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:14.863876 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d956e2b-c623-450b-88b7-9c0bf1beb105" containerName="revoke-keys" Jul 10 19:44:14.864017 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:14.863888 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d956e2b-c623-450b-88b7-9c0bf1beb105" containerName="revoke-keys" Jul 10 19:44:14.864017 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:14.863966 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d956e2b-c623-450b-88b7-9c0bf1beb105" containerName="revoke-keys" Jul 10 19:44:14.867246 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:14.867228 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7346-139701951232880" Jul 10 19:44:14.869511 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:14.869492 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-drjpt\"" Jul 10 19:44:14.875508 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:14.875476 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7346-139701951232880"] Jul 10 19:44:14.950000 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:14.949957 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89sc4\" (UniqueName: \"kubernetes.io/projected/22325e23-468e-4a3a-9c6e-735498531a97-kube-api-access-89sc4\") pod \"test-curl-7346-139701951232880\" (UID: \"22325e23-468e-4a3a-9c6e-735498531a97\") " pod="openshift-ingress/test-curl-7346-139701951232880" Jul 10 19:44:15.050709 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:15.050647 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89sc4\" (UniqueName: \"kubernetes.io/projected/22325e23-468e-4a3a-9c6e-735498531a97-kube-api-access-89sc4\") pod \"test-curl-7346-139701951232880\" (UID: \"22325e23-468e-4a3a-9c6e-735498531a97\") " pod="openshift-ingress/test-curl-7346-139701951232880" Jul 10 19:44:15.059627 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:15.059590 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89sc4\" (UniqueName: \"kubernetes.io/projected/22325e23-468e-4a3a-9c6e-735498531a97-kube-api-access-89sc4\") pod \"test-curl-7346-139701951232880\" (UID: \"22325e23-468e-4a3a-9c6e-735498531a97\") " pod="openshift-ingress/test-curl-7346-139701951232880" Jul 10 19:44:15.183637 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:15.183536 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7346-139701951232880" Jul 10 19:44:15.319734 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:15.319709 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7346-139701951232880"] Jul 10 19:44:15.321421 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:44:15.321393 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod22325e23_468e_4a3a_9c6e_735498531a97.slice/crio-1827dd76cd8a062dfb05b8c3d18f5d6d6b1bcc987a99062538f031b6d97c1834 WatchSource:0}: Error finding container 1827dd76cd8a062dfb05b8c3d18f5d6d6b1bcc987a99062538f031b6d97c1834: Status 404 returned error can't find the container with id 1827dd76cd8a062dfb05b8c3d18f5d6d6b1bcc987a99062538f031b6d97c1834 Jul 10 19:44:15.323060 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:15.323042 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 19:44:15.816459 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:15.816416 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7346-139701951232880" event={"ID":"22325e23-468e-4a3a-9c6e-735498531a97","Type":"ContainerStarted","Data":"1827dd76cd8a062dfb05b8c3d18f5d6d6b1bcc987a99062538f031b6d97c1834"} Jul 10 19:44:16.824116 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:16.824076 2581 generic.go:356] "Generic (PLEG): container finished" podID="22325e23-468e-4a3a-9c6e-735498531a97" containerID="0c1544404936a6f4e39a64fde24378fc9a5eafd461d867ed7f85321bd7ebf912" exitCode=0 Jul 10 19:44:16.824506 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:16.824128 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7346-139701951232880" event={"ID":"22325e23-468e-4a3a-9c6e-735498531a97","Type":"ContainerDied","Data":"0c1544404936a6f4e39a64fde24378fc9a5eafd461d867ed7f85321bd7ebf912"} Jul 10 19:44:17.967985 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:17.967953 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7346-139701951232880" Jul 10 19:44:18.005917 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:18.005850 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7346-139701951232880_22325e23-468e-4a3a-9c6e-735498531a97/test-curl-7346-139701951232880/0.log" Jul 10 19:44:18.028467 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:18.028438 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7346-139701951232880"] Jul 10 19:44:18.031029 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:18.031005 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7346-139701951232880"] Jul 10 19:44:18.077322 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:18.077290 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-89sc4\" (UniqueName: \"kubernetes.io/projected/22325e23-468e-4a3a-9c6e-735498531a97-kube-api-access-89sc4\") pod \"22325e23-468e-4a3a-9c6e-735498531a97\" (UID: \"22325e23-468e-4a3a-9c6e-735498531a97\") " Jul 10 19:44:18.079391 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:18.079361 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22325e23-468e-4a3a-9c6e-735498531a97-kube-api-access-89sc4" (OuterVolumeSpecName: "kube-api-access-89sc4") pod "22325e23-468e-4a3a-9c6e-735498531a97" (UID: "22325e23-468e-4a3a-9c6e-735498531a97"). InnerVolumeSpecName "kube-api-access-89sc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:44:18.147504 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:18.147470 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7346-139701951232880"] Jul 10 19:44:18.147951 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:18.147936 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22325e23-468e-4a3a-9c6e-735498531a97" containerName="test-curl-7346-139701951232880" Jul 10 19:44:18.148002 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:18.147953 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="22325e23-468e-4a3a-9c6e-735498531a97" containerName="test-curl-7346-139701951232880" Jul 10 19:44:18.148036 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:18.148030 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="22325e23-468e-4a3a-9c6e-735498531a97" containerName="test-curl-7346-139701951232880" Jul 10 19:44:18.151130 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:18.151114 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7346-139701951232880" Jul 10 19:44:18.155322 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:18.155291 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7346-139701951232880"] Jul 10 19:44:18.178886 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:18.178853 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89sc4\" (UniqueName: \"kubernetes.io/projected/22325e23-468e-4a3a-9c6e-735498531a97-kube-api-access-89sc4\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:44:18.279350 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:18.279312 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz9sr\" (UniqueName: \"kubernetes.io/projected/4b9cf2cd-1da9-4f0b-87ef-1ad03e6b8dc0-kube-api-access-lz9sr\") pod \"test-curl-7346-139701951232880\" (UID: \"4b9cf2cd-1da9-4f0b-87ef-1ad03e6b8dc0\") " pod="openshift-ingress/test-curl-7346-139701951232880" Jul 10 19:44:18.380034 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:18.379944 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz9sr\" (UniqueName: \"kubernetes.io/projected/4b9cf2cd-1da9-4f0b-87ef-1ad03e6b8dc0-kube-api-access-lz9sr\") pod \"test-curl-7346-139701951232880\" (UID: \"4b9cf2cd-1da9-4f0b-87ef-1ad03e6b8dc0\") " pod="openshift-ingress/test-curl-7346-139701951232880" Jul 10 19:44:18.389418 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:18.389380 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz9sr\" (UniqueName: \"kubernetes.io/projected/4b9cf2cd-1da9-4f0b-87ef-1ad03e6b8dc0-kube-api-access-lz9sr\") pod \"test-curl-7346-139701951232880\" (UID: \"4b9cf2cd-1da9-4f0b-87ef-1ad03e6b8dc0\") " pod="openshift-ingress/test-curl-7346-139701951232880" Jul 10 19:44:18.461280 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:18.461232 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7346-139701951232880" Jul 10 19:44:18.543394 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:18.543362 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22325e23-468e-4a3a-9c6e-735498531a97" path="/var/lib/kubelet/pods/22325e23-468e-4a3a-9c6e-735498531a97/volumes" Jul 10 19:44:18.604466 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:18.604439 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7346-139701951232880"] Jul 10 19:44:18.605728 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:44:18.605696 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b9cf2cd_1da9_4f0b_87ef_1ad03e6b8dc0.slice/crio-8ab1526dc103be756ea64c3ab4e23b1d0a4ac2477e5fccb25fdc50b31c2e0a5f WatchSource:0}: Error finding container 8ab1526dc103be756ea64c3ab4e23b1d0a4ac2477e5fccb25fdc50b31c2e0a5f: Status 404 returned error can't find the container with id 8ab1526dc103be756ea64c3ab4e23b1d0a4ac2477e5fccb25fdc50b31c2e0a5f Jul 10 19:44:18.837415 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:18.837384 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7346-139701951232880" Jul 10 19:44:18.837415 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:18.837404 2581 scope.go:117] "RemoveContainer" containerID="0c1544404936a6f4e39a64fde24378fc9a5eafd461d867ed7f85321bd7ebf912" Jul 10 19:44:18.838845 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:18.838816 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7346-139701951232880" event={"ID":"4b9cf2cd-1da9-4f0b-87ef-1ad03e6b8dc0","Type":"ContainerStarted","Data":"8ab1526dc103be756ea64c3ab4e23b1d0a4ac2477e5fccb25fdc50b31c2e0a5f"} Jul 10 19:44:19.847552 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:19.847507 2581 generic.go:356] "Generic (PLEG): container finished" podID="4b9cf2cd-1da9-4f0b-87ef-1ad03e6b8dc0" containerID="51533ff4bdc582c78f6baddfb42ce42e3ff450fab59e48fd732261d32e52f91d" exitCode=0 Jul 10 19:44:19.848010 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:19.847585 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7346-139701951232880" event={"ID":"4b9cf2cd-1da9-4f0b-87ef-1ad03e6b8dc0","Type":"ContainerDied","Data":"51533ff4bdc582c78f6baddfb42ce42e3ff450fab59e48fd732261d32e52f91d"} Jul 10 19:44:21.001155 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:21.001126 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7346-139701951232880" Jul 10 19:44:21.021839 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:21.021815 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7346-139701951232880_4b9cf2cd-1da9-4f0b-87ef-1ad03e6b8dc0/test-curl-7346-139701951232880/0.log" Jul 10 19:44:21.045675 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:21.045623 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7346-139701951232880"] Jul 10 19:44:21.048482 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:21.048455 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7346-139701951232880"] Jul 10 19:44:21.111172 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:21.111084 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lz9sr\" (UniqueName: \"kubernetes.io/projected/4b9cf2cd-1da9-4f0b-87ef-1ad03e6b8dc0-kube-api-access-lz9sr\") pod \"4b9cf2cd-1da9-4f0b-87ef-1ad03e6b8dc0\" (UID: \"4b9cf2cd-1da9-4f0b-87ef-1ad03e6b8dc0\") " Jul 10 19:44:21.113186 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:21.113155 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b9cf2cd-1da9-4f0b-87ef-1ad03e6b8dc0-kube-api-access-lz9sr" (OuterVolumeSpecName: "kube-api-access-lz9sr") pod "4b9cf2cd-1da9-4f0b-87ef-1ad03e6b8dc0" (UID: "4b9cf2cd-1da9-4f0b-87ef-1ad03e6b8dc0"). InnerVolumeSpecName "kube-api-access-lz9sr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:44:21.165319 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:21.165285 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7346-139701951233888"] Jul 10 19:44:21.165746 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:21.165730 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b9cf2cd-1da9-4f0b-87ef-1ad03e6b8dc0" containerName="test-curl-7346-139701951232880" Jul 10 19:44:21.165746 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:21.165746 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b9cf2cd-1da9-4f0b-87ef-1ad03e6b8dc0" containerName="test-curl-7346-139701951232880" Jul 10 19:44:21.165867 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:21.165805 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b9cf2cd-1da9-4f0b-87ef-1ad03e6b8dc0" containerName="test-curl-7346-139701951232880" Jul 10 19:44:21.168845 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:21.168828 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7346-139701951233888" Jul 10 19:44:21.177974 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:21.177949 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7346-139701951233888"] Jul 10 19:44:21.212301 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:21.212270 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz9sr\" (UniqueName: \"kubernetes.io/projected/4b9cf2cd-1da9-4f0b-87ef-1ad03e6b8dc0-kube-api-access-lz9sr\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:44:21.312929 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:21.312889 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt9p5\" (UniqueName: \"kubernetes.io/projected/0407e805-4533-4c64-9d78-ba81d1f152e1-kube-api-access-nt9p5\") pod \"test-curl-7346-139701951233888\" (UID: \"0407e805-4533-4c64-9d78-ba81d1f152e1\") " pod="openshift-ingress/test-curl-7346-139701951233888" Jul 10 19:44:21.413780 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:21.413652 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt9p5\" (UniqueName: \"kubernetes.io/projected/0407e805-4533-4c64-9d78-ba81d1f152e1-kube-api-access-nt9p5\") pod \"test-curl-7346-139701951233888\" (UID: \"0407e805-4533-4c64-9d78-ba81d1f152e1\") " pod="openshift-ingress/test-curl-7346-139701951233888" Jul 10 19:44:21.422221 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:21.422193 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt9p5\" (UniqueName: \"kubernetes.io/projected/0407e805-4533-4c64-9d78-ba81d1f152e1-kube-api-access-nt9p5\") pod \"test-curl-7346-139701951233888\" (UID: \"0407e805-4533-4c64-9d78-ba81d1f152e1\") " pod="openshift-ingress/test-curl-7346-139701951233888" Jul 10 19:44:21.478163 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:21.478132 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7346-139701951233888" Jul 10 19:44:21.616114 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:21.616087 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7346-139701951233888"] Jul 10 19:44:21.617396 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:44:21.617374 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0407e805_4533_4c64_9d78_ba81d1f152e1.slice/crio-f3af81c5774154529efe0b3defca9bc4ab65f57061dfd144680fe1e641bdec1e WatchSource:0}: Error finding container f3af81c5774154529efe0b3defca9bc4ab65f57061dfd144680fe1e641bdec1e: Status 404 returned error can't find the container with id f3af81c5774154529efe0b3defca9bc4ab65f57061dfd144680fe1e641bdec1e Jul 10 19:44:21.861084 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:21.861053 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7346-139701951232880" Jul 10 19:44:21.861250 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:21.861053 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ab1526dc103be756ea64c3ab4e23b1d0a4ac2477e5fccb25fdc50b31c2e0a5f" Jul 10 19:44:21.862361 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:21.862327 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7346-139701951233888" event={"ID":"0407e805-4533-4c64-9d78-ba81d1f152e1","Type":"ContainerStarted","Data":"f3af81c5774154529efe0b3defca9bc4ab65f57061dfd144680fe1e641bdec1e"} Jul 10 19:44:21.871625 ip-10-0-135-67 kubenswrapper[2581]: E0710 19:44:21.871599 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7346-139701951232880\" is forbidden: User \"system:node:ip-10-0-135-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-135-67.ec2.internal' and this object" podUID="4b9cf2cd-1da9-4f0b-87ef-1ad03e6b8dc0" pod="openshift-ingress/test-curl-7346-139701951232880" Jul 10 19:44:22.542870 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:22.542834 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b9cf2cd-1da9-4f0b-87ef-1ad03e6b8dc0" path="/var/lib/kubelet/pods/4b9cf2cd-1da9-4f0b-87ef-1ad03e6b8dc0/volumes" Jul 10 19:44:22.869848 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:22.869754 2581 generic.go:356] "Generic (PLEG): container finished" podID="0407e805-4533-4c64-9d78-ba81d1f152e1" containerID="42532f0e3f08c7f0c4a5fb08799e7add25e2001323e777fae3df45d6d61399b1" exitCode=0 Jul 10 19:44:22.869848 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:22.869833 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7346-139701951233888" event={"ID":"0407e805-4533-4c64-9d78-ba81d1f152e1","Type":"ContainerDied","Data":"42532f0e3f08c7f0c4a5fb08799e7add25e2001323e777fae3df45d6d61399b1"} Jul 10 19:44:24.024357 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:24.024330 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7346-139701951233888" Jul 10 19:44:24.044412 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:24.044344 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7346-139701951233888_0407e805-4533-4c64-9d78-ba81d1f152e1/test-curl-7346-139701951233888/0.log" Jul 10 19:44:24.068355 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:24.068311 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7346-139701951233888"] Jul 10 19:44:24.071950 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:24.071924 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7346-139701951233888"] Jul 10 19:44:24.135319 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:24.135222 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nt9p5\" (UniqueName: \"kubernetes.io/projected/0407e805-4533-4c64-9d78-ba81d1f152e1-kube-api-access-nt9p5\") pod \"0407e805-4533-4c64-9d78-ba81d1f152e1\" (UID: \"0407e805-4533-4c64-9d78-ba81d1f152e1\") " Jul 10 19:44:24.137409 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:24.137373 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0407e805-4533-4c64-9d78-ba81d1f152e1-kube-api-access-nt9p5" (OuterVolumeSpecName: "kube-api-access-nt9p5") pod "0407e805-4533-4c64-9d78-ba81d1f152e1" (UID: "0407e805-4533-4c64-9d78-ba81d1f152e1"). InnerVolumeSpecName "kube-api-access-nt9p5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:44:24.236302 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:24.236263 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt9p5\" (UniqueName: \"kubernetes.io/projected/0407e805-4533-4c64-9d78-ba81d1f152e1-kube-api-access-nt9p5\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:44:24.542954 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:24.542919 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0407e805-4533-4c64-9d78-ba81d1f152e1" path="/var/lib/kubelet/pods/0407e805-4533-4c64-9d78-ba81d1f152e1/volumes" Jul 10 19:44:24.883071 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:24.882978 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7346-139701951233888" Jul 10 19:44:24.883071 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:24.882995 2581 scope.go:117] "RemoveContainer" containerID="42532f0e3f08c7f0c4a5fb08799e7add25e2001323e777fae3df45d6d61399b1" Jul 10 19:44:26.951654 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:26.951609 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7207cb-62xhz"] Jul 10 19:44:26.952245 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:26.952223 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0407e805-4533-4c64-9d78-ba81d1f152e1" containerName="test-curl-7346-139701951233888" Jul 10 19:44:26.952327 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:26.952247 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0407e805-4533-4c64-9d78-ba81d1f152e1" containerName="test-curl-7346-139701951233888" Jul 10 19:44:26.952382 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:26.952352 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0407e805-4533-4c64-9d78-ba81d1f152e1" containerName="test-curl-7346-139701951233888" Jul 10 19:44:26.956995 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:26.956972 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7207cb-62xhz" Jul 10 19:44:26.959479 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:26.959353 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3723bbec-47ac-4531-b86a-f0aae2a80ef5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-7207cb-62xhz\" (UID: \"3723bbec-47ac-4531-b86a-f0aae2a80ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7207cb-62xhz" Jul 10 19:44:26.959785 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:26.959760 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fsnnw\"" Jul 10 19:44:26.962962 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:26.962942 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7207cb-62xhz"] Jul 10 19:44:27.060397 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:27.060362 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3723bbec-47ac-4531-b86a-f0aae2a80ef5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-7207cb-62xhz\" (UID: \"3723bbec-47ac-4531-b86a-f0aae2a80ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7207cb-62xhz" Jul 10 19:44:27.060985 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:27.060966 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3723bbec-47ac-4531-b86a-f0aae2a80ef5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-7207cb-62xhz\" (UID: \"3723bbec-47ac-4531-b86a-f0aae2a80ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7207cb-62xhz" Jul 10 19:44:27.269774 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:27.269736 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7207cb-62xhz" Jul 10 19:44:27.415603 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:27.415575 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7207cb-62xhz"] Jul 10 19:44:27.417452 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:44:27.417421 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3723bbec_47ac_4531_b86a_f0aae2a80ef5.slice/crio-486c366c6a7372f981fa091c0ec8257648aec5c918c838de8685801c60f17cfb WatchSource:0}: Error finding container 486c366c6a7372f981fa091c0ec8257648aec5c918c838de8685801c60f17cfb: Status 404 returned error can't find the container with id 486c366c6a7372f981fa091c0ec8257648aec5c918c838de8685801c60f17cfb Jul 10 19:44:27.906408 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:27.906308 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7207cb-62xhz" event={"ID":"3723bbec-47ac-4531-b86a-f0aae2a80ef5","Type":"ContainerStarted","Data":"454981ebe514fb13157f0c48c59b7458fb43f2f54019c4ad221be4c2ae4a4661"} Jul 10 19:44:27.906408 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:27.906402 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7207cb-62xhz" event={"ID":"3723bbec-47ac-4531-b86a-f0aae2a80ef5","Type":"ContainerStarted","Data":"486c366c6a7372f981fa091c0ec8257648aec5c918c838de8685801c60f17cfb"} Jul 10 19:44:27.942111 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:27.942061 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7207cb-62xhz" podStartSLOduration=1.9420462 podStartE2EDuration="1.9420462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:44:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:44:27.940350589 +0000 UTC m=+2751.984493096" watchObservedRunningTime="2026-07-10 19:44:27.9420462 +0000 UTC m=+2751.986188701" Jul 10 19:44:28.914746 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:28.914671 2581 generic.go:356] "Generic (PLEG): container finished" podID="3723bbec-47ac-4531-b86a-f0aae2a80ef5" containerID="454981ebe514fb13157f0c48c59b7458fb43f2f54019c4ad221be4c2ae4a4661" exitCode=0 Jul 10 19:44:28.915153 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:28.914716 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7207cb-62xhz" event={"ID":"3723bbec-47ac-4531-b86a-f0aae2a80ef5","Type":"ContainerDied","Data":"454981ebe514fb13157f0c48c59b7458fb43f2f54019c4ad221be4c2ae4a4661"} Jul 10 19:44:30.065532 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:30.065502 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7207cb-62xhz" Jul 10 19:44:30.078706 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:30.078662 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3723bbec-47ac-4531-b86a-f0aae2a80ef5-maas-api-service-ca\") pod \"3723bbec-47ac-4531-b86a-f0aae2a80ef5\" (UID: \"3723bbec-47ac-4531-b86a-f0aae2a80ef5\") " Jul 10 19:44:30.079019 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:30.078996 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3723bbec-47ac-4531-b86a-f0aae2a80ef5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3723bbec-47ac-4531-b86a-f0aae2a80ef5" (UID: "3723bbec-47ac-4531-b86a-f0aae2a80ef5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:44:30.180044 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:30.179946 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3723bbec-47ac-4531-b86a-f0aae2a80ef5-maas-api-service-ca\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:44:30.928573 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:30.928491 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7207cb-62xhz" Jul 10 19:44:30.928573 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:30.928524 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7207cb-62xhz" event={"ID":"3723bbec-47ac-4531-b86a-f0aae2a80ef5","Type":"ContainerDied","Data":"486c366c6a7372f981fa091c0ec8257648aec5c918c838de8685801c60f17cfb"} Jul 10 19:44:30.928573 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:30.928575 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="486c366c6a7372f981fa091c0ec8257648aec5c918c838de8685801c60f17cfb" Jul 10 19:44:53.044366 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:53.044328 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3ec3c4-ws6vr"] Jul 10 19:44:53.044906 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:53.044740 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3723bbec-47ac-4531-b86a-f0aae2a80ef5" containerName="revoke-keys" Jul 10 19:44:53.044906 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:53.044775 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3723bbec-47ac-4531-b86a-f0aae2a80ef5" containerName="revoke-keys" Jul 10 19:44:53.044906 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:53.044865 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3723bbec-47ac-4531-b86a-f0aae2a80ef5" containerName="revoke-keys" Jul 10 19:44:53.050034 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:53.050015 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3ec3c4-ws6vr" Jul 10 19:44:53.052508 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:53.052485 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fsnnw\"" Jul 10 19:44:53.070130 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:53.070095 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3ec3c4-ws6vr"] Jul 10 19:44:53.177643 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:53.177594 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63433535-eef9-450e-ba99-42afb53fdd05-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-3ec3c4-ws6vr\" (UID: \"63433535-eef9-450e-ba99-42afb53fdd05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3ec3c4-ws6vr" Jul 10 19:44:53.278066 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:53.278027 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63433535-eef9-450e-ba99-42afb53fdd05-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-3ec3c4-ws6vr\" (UID: \"63433535-eef9-450e-ba99-42afb53fdd05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3ec3c4-ws6vr" Jul 10 19:44:53.278672 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:53.278650 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63433535-eef9-450e-ba99-42afb53fdd05-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-3ec3c4-ws6vr\" (UID: \"63433535-eef9-450e-ba99-42afb53fdd05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3ec3c4-ws6vr" Jul 10 19:44:53.372920 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:53.372832 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3ec3c4-ws6vr" Jul 10 19:44:53.522196 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:53.522158 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3ec3c4-ws6vr"] Jul 10 19:44:53.523391 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:44:53.523366 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63433535_eef9_450e_ba99_42afb53fdd05.slice/crio-04c5dd401cec0f3fe0403383de5eeeea4cfcce46637acb75a045378eb0c2b01b WatchSource:0}: Error finding container 04c5dd401cec0f3fe0403383de5eeeea4cfcce46637acb75a045378eb0c2b01b: Status 404 returned error can't find the container with id 04c5dd401cec0f3fe0403383de5eeeea4cfcce46637acb75a045378eb0c2b01b Jul 10 19:44:54.071630 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:54.071586 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3ec3c4-ws6vr" event={"ID":"63433535-eef9-450e-ba99-42afb53fdd05","Type":"ContainerStarted","Data":"a22c235ebde0a63dc3bb95232bc7151d7e355a607514874d83a27537c504e59b"} Jul 10 19:44:54.072064 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:54.071650 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3ec3c4-ws6vr" event={"ID":"63433535-eef9-450e-ba99-42afb53fdd05","Type":"ContainerStarted","Data":"04c5dd401cec0f3fe0403383de5eeeea4cfcce46637acb75a045378eb0c2b01b"} Jul 10 19:44:54.088651 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:54.088601 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3ec3c4-ws6vr" podStartSLOduration=1.088585987 podStartE2EDuration="1.088585987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:44:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:44:54.086319842 +0000 UTC m=+2778.130462393" watchObservedRunningTime="2026-07-10 19:44:54.088585987 +0000 UTC m=+2778.132728527" Jul 10 19:44:55.079584 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:55.079547 2581 generic.go:356] "Generic (PLEG): container finished" podID="63433535-eef9-450e-ba99-42afb53fdd05" containerID="a22c235ebde0a63dc3bb95232bc7151d7e355a607514874d83a27537c504e59b" exitCode=0 Jul 10 19:44:55.079974 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:55.079632 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3ec3c4-ws6vr" event={"ID":"63433535-eef9-450e-ba99-42afb53fdd05","Type":"ContainerDied","Data":"a22c235ebde0a63dc3bb95232bc7151d7e355a607514874d83a27537c504e59b"} Jul 10 19:44:56.222886 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:56.222856 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3ec3c4-ws6vr" Jul 10 19:44:56.308817 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:56.308770 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63433535-eef9-450e-ba99-42afb53fdd05-maas-api-service-ca\") pod \"63433535-eef9-450e-ba99-42afb53fdd05\" (UID: \"63433535-eef9-450e-ba99-42afb53fdd05\") " Jul 10 19:44:56.309150 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:56.309127 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63433535-eef9-450e-ba99-42afb53fdd05-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "63433535-eef9-450e-ba99-42afb53fdd05" (UID: "63433535-eef9-450e-ba99-42afb53fdd05"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:44:56.410115 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:56.410023 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63433535-eef9-450e-ba99-42afb53fdd05-maas-api-service-ca\") on node \"ip-10-0-135-67.ec2.internal\" DevicePath \"\"" Jul 10 19:44:57.092969 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:57.092939 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3ec3c4-ws6vr" Jul 10 19:44:57.093168 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:57.092970 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3ec3c4-ws6vr" event={"ID":"63433535-eef9-450e-ba99-42afb53fdd05","Type":"ContainerDied","Data":"04c5dd401cec0f3fe0403383de5eeeea4cfcce46637acb75a045378eb0c2b01b"} Jul 10 19:44:57.093168 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:44:57.093016 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04c5dd401cec0f3fe0403383de5eeeea4cfcce46637acb75a045378eb0c2b01b" Jul 10 19:45:00.147409 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:00.147323 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3ec3c4-29728545-dcvdl"] Jul 10 19:45:00.147801 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:00.147773 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63433535-eef9-450e-ba99-42afb53fdd05" containerName="revoke-keys" Jul 10 19:45:00.147801 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:00.147786 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="63433535-eef9-450e-ba99-42afb53fdd05" containerName="revoke-keys" Jul 10 19:45:00.147873 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:00.147866 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="63433535-eef9-450e-ba99-42afb53fdd05" containerName="revoke-keys" Jul 10 19:45:00.151146 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:00.151129 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3ec3c4-29728545-dcvdl" Jul 10 19:45:00.153661 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:00.153644 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fsnnw\"" Jul 10 19:45:00.161183 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:00.161159 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3ec3c4-29728545-dcvdl" Jul 10 19:45:00.164990 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:00.164069 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3ec3c4-29728545-dcvdl"] Jul 10 19:45:00.300266 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:00.300240 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3ec3c4-29728545-dcvdl"] Jul 10 19:45:00.303251 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:45:00.303214 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4393fef_fa93_46f2_9248_800b5d1e25c4.slice/crio-8288cd685ab65d1cc88af9b60195492e5d67a5e9e5380de0c6d7cdc097d2f26a WatchSource:0}: Error finding container 8288cd685ab65d1cc88af9b60195492e5d67a5e9e5380de0c6d7cdc097d2f26a: Status 404 returned error can't find the container with id 8288cd685ab65d1cc88af9b60195492e5d67a5e9e5380de0c6d7cdc097d2f26a Jul 10 19:45:01.119389 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:01.119354 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3ec3c4-29728545-dcvdl" event={"ID":"c4393fef-fa93-46f2-9248-800b5d1e25c4","Type":"ContainerStarted","Data":"81ceeee98c96131b6d90761857195154609b5d1b4b1463cce51c62e221a65d24"} Jul 10 19:45:01.119611 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:01.119406 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3ec3c4-29728545-dcvdl" event={"ID":"c4393fef-fa93-46f2-9248-800b5d1e25c4","Type":"ContainerStarted","Data":"8288cd685ab65d1cc88af9b60195492e5d67a5e9e5380de0c6d7cdc097d2f26a"} Jul 10 19:45:01.136316 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:01.136259 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3ec3c4-29728545-dcvdl" podStartSLOduration=1.136242365 podStartE2EDuration="1.136242365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:45:01.134316665 +0000 UTC m=+2785.178459188" watchObservedRunningTime="2026-07-10 19:45:01.136242365 +0000 UTC m=+2785.180384867" Jul 10 19:45:02.127046 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:02.127008 2581 generic.go:356] "Generic (PLEG): container finished" podID="c4393fef-fa93-46f2-9248-800b5d1e25c4" containerID="81ceeee98c96131b6d90761857195154609b5d1b4b1463cce51c62e221a65d24" exitCode=0 Jul 10 19:45:02.127046 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:02.127052 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3ec3c4-29728545-dcvdl" event={"ID":"c4393fef-fa93-46f2-9248-800b5d1e25c4","Type":"ContainerDied","Data":"81ceeee98c96131b6d90761857195154609b5d1b4b1463cce51c62e221a65d24"} Jul 10 19:45:03.274137 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:03.274102 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3ec3c4-29728545-dcvdl" Jul 10 19:45:03.530188 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:03.529237 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3ec3c4-29728545-dcvdl"] Jul 10 19:45:03.539231 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:03.539194 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3ec3c4-29728545-dcvdl"] Jul 10 19:45:04.141448 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:04.141418 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3ec3c4-29728545-dcvdl" Jul 10 19:45:04.141448 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:04.141449 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8288cd685ab65d1cc88af9b60195492e5d67a5e9e5380de0c6d7cdc097d2f26a" Jul 10 19:45:04.151443 ip-10-0-135-67 kubenswrapper[2581]: E0710 19:45:04.151413 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"maas-api-key-cleanup-e2e-shared-b-3ec3c4-29728545-dcvdl\" is forbidden: User \"system:node:ip-10-0-135-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-135-67.ec2.internal' and this object" podUID="c4393fef-fa93-46f2-9248-800b5d1e25c4" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3ec3c4-29728545-dcvdl" Jul 10 19:45:04.542765 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:04.542734 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4393fef-fa93-46f2-9248-800b5d1e25c4" path="/var/lib/kubelet/pods/c4393fef-fa93-46f2-9248-800b5d1e25c4/volumes" Jul 10 19:45:24.067174 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:24.067136 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-b4f45_112b1d22-d1b8-4de9-9baf-2821f4566806/manager/0.log" Jul 10 19:45:24.191998 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:24.191961 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-kkqx7_57d9470d-fca4-4f43-907d-3b40595570fb/manager/0.log" Jul 10 19:45:24.315393 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:24.315355 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-77d7f68d48-hnrxv_2e1f0282-7303-43da-ae3c-767a14ab8c16/manager/0.log" Jul 10 19:45:24.423625 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:24.423543 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-92hm5_249c9d5b-dc25-47cb-b492-cb0ca0805ea9/server/0.log" Jul 10 19:45:24.546493 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:24.546463 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-9g4cr_29957d41-1783-4280-bee2-75c4529ab426/manager/0.log" Jul 10 19:45:24.892840 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:24.892810 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7cfcf59dd-wwbcb_3c64a97c-e4f5-49d7-ad20-18a4ec847644/copy-manifests/0.log" Jul 10 19:45:24.911576 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:24.911540 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7cfcf59dd-wwbcb_3c64a97c-e4f5-49d7-ad20-18a4ec847644/manager/0.log" Jul 10 19:45:25.267184 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:25.267151 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6789bd89c-sg8d5_143f3fb1-7601-4826-b1b6-6eddca57b03e/maas-api/0.log" Jul 10 19:45:25.685817 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:25.685736 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-6402da18-4vlzq_d04de4d1-36b2-4d1e-b95b-3f5304ca217e/revoke-keys/0.log" Jul 10 19:45:25.789976 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:25.789943 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-6e61e5c8-nv64n_a9136b28-7347-4f29-9070-f367c49a40b9/revoke-keys/0.log" Jul 10 19:45:25.901732 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:25.901705 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-a-9d1921-wd9fx_fbe338d5-3c79-4bc2-a33b-07a7d5037493/revoke-keys/0.log" Jul 10 19:45:26.002829 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:26.002798 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-e78db6-r4gfj_7d956e2b-c623-450b-88b7-9c0bf1beb105/revoke-keys/0.log" Jul 10 19:45:26.108074 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:26.108045 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-derive-2397a366-ww45x_ad749dff-107e-4587-a8f8-4199fd77971e/revoke-keys/0.log" Jul 10 19:45:26.220501 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:26.220461 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-a-fd22f1-v7cck_0317525e-4866-4139-a490-1a540ab90893/revoke-keys/0.log" Jul 10 19:45:26.328115 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:26.328027 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-b-8c42c2-5smpw_c2bfe8ce-f4a4-4620-b3a1-ac3a2e0ecafa/revoke-keys/0.log" Jul 10 19:45:26.430942 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:26.430915 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-7d2414db-q2clr_dca83456-8418-41ac-8205-bdfb27251795/revoke-keys/0.log" Jul 10 19:45:26.540656 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:26.540625 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-cacc33b0-frmhx_3d366571-e21b-48d5-b355-7085715b7e8f/revoke-keys/0.log" Jul 10 19:45:26.649965 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:26.649883 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-a-7207cb-62xhz_3723bbec-47ac-4531-b86a-f0aae2a80ef5/revoke-keys/0.log" Jul 10 19:45:26.759001 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:26.758972 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-b-3ec3c4-ws6vr_63433535-eef9-450e-ba99-42afb53fdd05/revoke-keys/0.log" Jul 10 19:45:26.864478 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:26.864449 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-rzrns_3b5943f0-10c9-4290-8710-a28dd4fd3aab/postgres/0.log" Jul 10 19:45:27.598823 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:27.598797 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gx4nh_8b0dd963-9313-419e-a83c-72908d524086/extract/0.log" Jul 10 19:45:27.611059 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:27.611022 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gx4nh_8b0dd963-9313-419e-a83c-72908d524086/util/0.log" Jul 10 19:45:27.617175 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:27.617144 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gx4nh_8b0dd963-9313-419e-a83c-72908d524086/pull/0.log" Jul 10 19:45:27.729576 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:27.729545 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nz55p_ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc/util/0.log" Jul 10 19:45:27.738434 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:27.738407 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nz55p_ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc/pull/0.log" Jul 10 19:45:27.745002 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:27.744978 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nz55p_ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc/extract/0.log" Jul 10 19:45:27.847938 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:27.847911 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tx75x_277dce86-11f7-4dc7-8209-c2021dfb69f2/util/0.log" Jul 10 19:45:27.856724 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:27.856633 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tx75x_277dce86-11f7-4dc7-8209-c2021dfb69f2/pull/0.log" Jul 10 19:45:27.863988 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:27.863969 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tx75x_277dce86-11f7-4dc7-8209-c2021dfb69f2/extract/0.log" Jul 10 19:45:27.974782 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:27.974749 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef156snh_ec050446-3850-4b65-bcf8-2e7bf2688638/util/0.log" Jul 10 19:45:27.980719 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:27.980672 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef156snh_ec050446-3850-4b65-bcf8-2e7bf2688638/pull/0.log" Jul 10 19:45:27.985956 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:27.985935 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef156snh_ec050446-3850-4b65-bcf8-2e7bf2688638/extract/0.log" Jul 10 19:45:28.213328 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:28.213241 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-hmfrt_4df8aa21-bb25-467b-b0c2-6392ff510974/manager/0.log" Jul 10 19:45:28.317901 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:28.317864 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-bs99p_f30bbd65-1ec6-44df-80ff-15b3e3e32b7b/manager/0.log" Jul 10 19:45:28.424251 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:28.424225 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-qtvk4_b260e5a5-56f3-47eb-9875-c2d0f740782a/kuadrant-console-plugin/0.log" Jul 10 19:45:28.545579 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:28.545546 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-wwz8v_670aa800-3f0d-4c74-830f-ac82faad07c9/registry-server/0.log" Jul 10 19:45:28.668584 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:28.668549 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-6pxgk_88426ec8-8c7e-4f16-b4c8-7ba42a36d8ac/manager/0.log" Jul 10 19:45:28.890611 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:28.890522 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-kr8fm_3c6823d8-1cae-4102-89db-5c7d8e30d7aa/manager/0.log" Jul 10 19:45:29.431474 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:29.431445 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-697457d895-hj6vs_d275207d-476a-4ce4-9ce2-da76a36e658d/kube-auth-proxy/0.log" Jul 10 19:45:29.990300 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:29.990271 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8c8d8cb74-pbqx5_348b64e9-4db6-4d74-9a29-36436b02e1d3/router/0.log" Jul 10 19:45:30.318247 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:30.318213 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-v7n6p_2836da0c-b392-4604-a10b-13a9e7b489fe/main/0.log" Jul 10 19:45:30.531029 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:30.530994 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-zq4w6_63d0e5fe-2d13-40dd-9bbc-46dda9b19320/main/0.log" Jul 10 19:45:30.760893 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:30.760853 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-g22tg_da3fe636-22e0-4a46-b9a5-49653ba2b691/main/0.log" Jul 10 19:45:37.367841 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:37.367794 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qc4kd_f7d3d4a3-cb7b-4737-8f95-e3b5aac1b5fe/global-pull-secret-syncer/0.log" Jul 10 19:45:37.546642 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:37.546608 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zlj4w_2c842677-9d8d-4210-9a3b-ff32bf8bda1e/konnectivity-agent/0.log" Jul 10 19:45:37.568898 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:37.568865 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-67.ec2.internal_e2729654f7488332f5966486449c5669/haproxy/0.log" Jul 10 19:45:42.184646 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:42.184610 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gx4nh_8b0dd963-9313-419e-a83c-72908d524086/extract/0.log" Jul 10 19:45:42.214944 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:42.214859 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gx4nh_8b0dd963-9313-419e-a83c-72908d524086/util/0.log" Jul 10 19:45:42.250882 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:42.250846 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gx4nh_8b0dd963-9313-419e-a83c-72908d524086/pull/0.log" Jul 10 19:45:42.276524 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:42.276496 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nz55p_ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc/extract/0.log" Jul 10 19:45:42.305407 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:42.305368 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nz55p_ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc/util/0.log" Jul 10 19:45:42.332488 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:42.332453 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nz55p_ee1a3f74-1040-4f73-a7d5-904fd9b2c9bc/pull/0.log" Jul 10 19:45:42.365054 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:42.365027 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tx75x_277dce86-11f7-4dc7-8209-c2021dfb69f2/extract/0.log" Jul 10 19:45:42.392549 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:42.392519 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tx75x_277dce86-11f7-4dc7-8209-c2021dfb69f2/util/0.log" Jul 10 19:45:42.421171 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:42.421143 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tx75x_277dce86-11f7-4dc7-8209-c2021dfb69f2/pull/0.log" Jul 10 19:45:42.454896 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:42.454869 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef156snh_ec050446-3850-4b65-bcf8-2e7bf2688638/extract/0.log" Jul 10 19:45:42.483391 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:42.483314 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef156snh_ec050446-3850-4b65-bcf8-2e7bf2688638/util/0.log" Jul 10 19:45:42.509041 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:42.509011 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef156snh_ec050446-3850-4b65-bcf8-2e7bf2688638/pull/0.log" Jul 10 19:45:42.986172 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:42.986137 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-hmfrt_4df8aa21-bb25-467b-b0c2-6392ff510974/manager/0.log" Jul 10 19:45:43.013182 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:43.013154 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-bs99p_f30bbd65-1ec6-44df-80ff-15b3e3e32b7b/manager/0.log" Jul 10 19:45:43.046492 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:43.046468 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-qtvk4_b260e5a5-56f3-47eb-9875-c2d0f740782a/kuadrant-console-plugin/0.log" Jul 10 19:45:43.109286 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:43.109249 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-wwz8v_670aa800-3f0d-4c74-830f-ac82faad07c9/registry-server/0.log" Jul 10 19:45:43.310185 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:43.310148 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-6pxgk_88426ec8-8c7e-4f16-b4c8-7ba42a36d8ac/manager/0.log" Jul 10 19:45:43.459881 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:43.459837 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-kr8fm_3c6823d8-1cae-4102-89db-5c7d8e30d7aa/manager/0.log" Jul 10 19:45:45.468816 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:45.468785 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-ftv8l_9769ce8f-d048-4fd4-b4dd-3a2d636859f9/monitoring-plugin/0.log" Jul 10 19:45:45.662654 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:45.662618 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p9h5z_9c52bdca-eacb-4bb6-9305-ada12d71dede/node-exporter/0.log" Jul 10 19:45:45.684014 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:45.683982 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p9h5z_9c52bdca-eacb-4bb6-9305-ada12d71dede/kube-rbac-proxy/0.log" Jul 10 19:45:45.705009 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:45.704983 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p9h5z_9c52bdca-eacb-4bb6-9305-ada12d71dede/init-textfile/0.log" Jul 10 19:45:46.003102 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:46.003058 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-d5ww8_d6eebd0a-7281-40c4-81d0-9d1088043c1d/prometheus-operator/0.log" Jul 10 19:45:46.023271 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:46.023237 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-d5ww8_d6eebd0a-7281-40c4-81d0-9d1088043c1d/kube-rbac-proxy/0.log" Jul 10 19:45:46.164079 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:46.164050 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f747dbb6f-gjdd7_3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85/thanos-query/0.log" Jul 10 19:45:46.189312 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:46.189278 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f747dbb6f-gjdd7_3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85/kube-rbac-proxy-web/0.log" Jul 10 19:45:46.217587 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:46.217553 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f747dbb6f-gjdd7_3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85/kube-rbac-proxy/0.log" Jul 10 19:45:46.243164 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:46.243136 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f747dbb6f-gjdd7_3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85/prom-label-proxy/0.log" Jul 10 19:45:46.266660 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:46.266567 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f747dbb6f-gjdd7_3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85/kube-rbac-proxy-rules/0.log" Jul 10 19:45:46.290939 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:46.290905 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f747dbb6f-gjdd7_3edc6eb7-d84c-4a16-bc7a-ea3bf88d3b85/kube-rbac-proxy-metrics/0.log" Jul 10 19:45:47.006458 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:47.006422 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-q9l4b/perf-node-gather-daemonset-cpc25"] Jul 10 19:45:47.007001 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:47.006980 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4393fef-fa93-46f2-9248-800b5d1e25c4" containerName="cleanup" Jul 10 19:45:47.007071 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:47.007004 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4393fef-fa93-46f2-9248-800b5d1e25c4" containerName="cleanup" Jul 10 19:45:47.007107 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:47.007076 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4393fef-fa93-46f2-9248-800b5d1e25c4" containerName="cleanup" Jul 10 19:45:47.010200 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:47.010183 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q9l4b/perf-node-gather-daemonset-cpc25" Jul 10 19:45:47.012920 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:47.012897 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q9l4b\"/\"openshift-service-ca.crt\"" Jul 10 19:45:47.013329 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:47.013312 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-q9l4b\"/\"default-dockercfg-7kvh2\"" Jul 10 19:45:47.013917 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:47.013899 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q9l4b\"/\"kube-root-ca.crt\"" Jul 10 19:45:47.021185 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:47.021161 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q9l4b/perf-node-gather-daemonset-cpc25"] Jul 10 19:45:47.068270 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:47.068225 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e7fdcfcf-b05d-4547-a814-b145e1d4fdf9-lib-modules\") pod \"perf-node-gather-daemonset-cpc25\" (UID: \"e7fdcfcf-b05d-4547-a814-b145e1d4fdf9\") " pod="openshift-must-gather-q9l4b/perf-node-gather-daemonset-cpc25" Jul 10 19:45:47.068270 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:47.068270 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmrv2\" (UniqueName: \"kubernetes.io/projected/e7fdcfcf-b05d-4547-a814-b145e1d4fdf9-kube-api-access-gmrv2\") pod \"perf-node-gather-daemonset-cpc25\" (UID: \"e7fdcfcf-b05d-4547-a814-b145e1d4fdf9\") " pod="openshift-must-gather-q9l4b/perf-node-gather-daemonset-cpc25" Jul 10 19:45:47.068508 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:47.068301 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e7fdcfcf-b05d-4547-a814-b145e1d4fdf9-podres\") pod \"perf-node-gather-daemonset-cpc25\" (UID: \"e7fdcfcf-b05d-4547-a814-b145e1d4fdf9\") " pod="openshift-must-gather-q9l4b/perf-node-gather-daemonset-cpc25" Jul 10 19:45:47.068508 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:47.068355 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e7fdcfcf-b05d-4547-a814-b145e1d4fdf9-proc\") pod \"perf-node-gather-daemonset-cpc25\" (UID: \"e7fdcfcf-b05d-4547-a814-b145e1d4fdf9\") " pod="openshift-must-gather-q9l4b/perf-node-gather-daemonset-cpc25" Jul 10 19:45:47.068508 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:47.068410 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e7fdcfcf-b05d-4547-a814-b145e1d4fdf9-sys\") pod \"perf-node-gather-daemonset-cpc25\" (UID: \"e7fdcfcf-b05d-4547-a814-b145e1d4fdf9\") " pod="openshift-must-gather-q9l4b/perf-node-gather-daemonset-cpc25" Jul 10 19:45:47.169641 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:47.169599 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e7fdcfcf-b05d-4547-a814-b145e1d4fdf9-podres\") pod \"perf-node-gather-daemonset-cpc25\" (UID: \"e7fdcfcf-b05d-4547-a814-b145e1d4fdf9\") " pod="openshift-must-gather-q9l4b/perf-node-gather-daemonset-cpc25" Jul 10 19:45:47.169872 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:47.169656 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e7fdcfcf-b05d-4547-a814-b145e1d4fdf9-proc\") pod \"perf-node-gather-daemonset-cpc25\" (UID: \"e7fdcfcf-b05d-4547-a814-b145e1d4fdf9\") " pod="openshift-must-gather-q9l4b/perf-node-gather-daemonset-cpc25" Jul 10 19:45:47.169872 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:47.169746 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e7fdcfcf-b05d-4547-a814-b145e1d4fdf9-sys\") pod \"perf-node-gather-daemonset-cpc25\" (UID: \"e7fdcfcf-b05d-4547-a814-b145e1d4fdf9\") " pod="openshift-must-gather-q9l4b/perf-node-gather-daemonset-cpc25" Jul 10 19:45:47.169872 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:47.169773 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e7fdcfcf-b05d-4547-a814-b145e1d4fdf9-lib-modules\") pod \"perf-node-gather-daemonset-cpc25\" (UID: \"e7fdcfcf-b05d-4547-a814-b145e1d4fdf9\") " pod="openshift-must-gather-q9l4b/perf-node-gather-daemonset-cpc25" Jul 10 19:45:47.169872 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:47.169789 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmrv2\" (UniqueName: \"kubernetes.io/projected/e7fdcfcf-b05d-4547-a814-b145e1d4fdf9-kube-api-access-gmrv2\") pod \"perf-node-gather-daemonset-cpc25\" (UID: \"e7fdcfcf-b05d-4547-a814-b145e1d4fdf9\") " pod="openshift-must-gather-q9l4b/perf-node-gather-daemonset-cpc25" Jul 10 19:45:47.169872 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:47.169798 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e7fdcfcf-b05d-4547-a814-b145e1d4fdf9-podres\") pod \"perf-node-gather-daemonset-cpc25\" (UID: \"e7fdcfcf-b05d-4547-a814-b145e1d4fdf9\") " pod="openshift-must-gather-q9l4b/perf-node-gather-daemonset-cpc25" Jul 10 19:45:47.169872 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:47.169835 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e7fdcfcf-b05d-4547-a814-b145e1d4fdf9-proc\") pod \"perf-node-gather-daemonset-cpc25\" (UID: \"e7fdcfcf-b05d-4547-a814-b145e1d4fdf9\") " pod="openshift-must-gather-q9l4b/perf-node-gather-daemonset-cpc25" Jul 10 19:45:47.169872 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:47.169845 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e7fdcfcf-b05d-4547-a814-b145e1d4fdf9-sys\") pod \"perf-node-gather-daemonset-cpc25\" (UID: \"e7fdcfcf-b05d-4547-a814-b145e1d4fdf9\") " pod="openshift-must-gather-q9l4b/perf-node-gather-daemonset-cpc25" Jul 10 19:45:47.170143 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:47.169881 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e7fdcfcf-b05d-4547-a814-b145e1d4fdf9-lib-modules\") pod \"perf-node-gather-daemonset-cpc25\" (UID: \"e7fdcfcf-b05d-4547-a814-b145e1d4fdf9\") " pod="openshift-must-gather-q9l4b/perf-node-gather-daemonset-cpc25" Jul 10 19:45:47.178848 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:47.178825 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmrv2\" (UniqueName: \"kubernetes.io/projected/e7fdcfcf-b05d-4547-a814-b145e1d4fdf9-kube-api-access-gmrv2\") pod \"perf-node-gather-daemonset-cpc25\" (UID: \"e7fdcfcf-b05d-4547-a814-b145e1d4fdf9\") " pod="openshift-must-gather-q9l4b/perf-node-gather-daemonset-cpc25" Jul 10 19:45:47.320718 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:47.320598 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q9l4b/perf-node-gather-daemonset-cpc25" Jul 10 19:45:47.469455 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:47.469427 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q9l4b/perf-node-gather-daemonset-cpc25"] Jul 10 19:45:47.470864 ip-10-0-135-67 kubenswrapper[2581]: W0710 19:45:47.470840 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode7fdcfcf_b05d_4547_a814_b145e1d4fdf9.slice/crio-96d9008ba8c463cb006725ca987783325f6e6ce63c7954161cd979d59328c519 WatchSource:0}: Error finding container 96d9008ba8c463cb006725ca987783325f6e6ce63c7954161cd979d59328c519: Status 404 returned error can't find the container with id 96d9008ba8c463cb006725ca987783325f6e6ce63c7954161cd979d59328c519 Jul 10 19:45:48.412333 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:48.412289 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q9l4b/perf-node-gather-daemonset-cpc25" event={"ID":"e7fdcfcf-b05d-4547-a814-b145e1d4fdf9","Type":"ContainerStarted","Data":"8955199e534065e8429ded0a0da57125c003a48784d297a1bdca6c3d36d2da6f"} Jul 10 19:45:48.412727 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:48.412354 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q9l4b/perf-node-gather-daemonset-cpc25" event={"ID":"e7fdcfcf-b05d-4547-a814-b145e1d4fdf9","Type":"ContainerStarted","Data":"96d9008ba8c463cb006725ca987783325f6e6ce63c7954161cd979d59328c519"} Jul 10 19:45:48.412727 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:48.412387 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-q9l4b/perf-node-gather-daemonset-cpc25" Jul 10 19:45:48.431842 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:48.431795 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-q9l4b/perf-node-gather-daemonset-cpc25" podStartSLOduration=2.431778619 podStartE2EDuration="2.431778619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 19:45:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:45:48.429709345 +0000 UTC m=+2832.473851860" watchObservedRunningTime="2026-07-10 19:45:48.431778619 +0000 UTC m=+2832.475921120" Jul 10 19:45:48.583504 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:48.583454 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-rhf5g_ad100d8d-a2d8-4c86-8b2c-2aecee40bfbf/download-server/0.log" Jul 10 19:45:49.171585 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:49.171554 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-68z78_1f90ac19-a36e-4ee5-ba73-c7f06f7950f9/volume-data-source-validator/0.log" Jul 10 19:45:50.334014 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:50.333983 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-v5rql_ed9ac220-276d-4864-b19b-17b8f69eaa98/dns/0.log" Jul 10 19:45:50.356851 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:50.356822 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-v5rql_ed9ac220-276d-4864-b19b-17b8f69eaa98/kube-rbac-proxy/0.log" Jul 10 19:45:50.437147 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:50.437117 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zf82d_df38c8b2-26d6-46bb-95ca-166ff4723ceb/dns-node-resolver/0.log" Jul 10 19:45:51.042768 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:51.042739 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-65859cd4cb-2n5rm_f8c1fa0c-a239-48b9-8dae-f94be920ae1a/registry/0.log" Jul 10 19:45:51.065974 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:51.065948 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7btwf_10a5a151-ab9d-45db-8502-26d00ec21659/node-ca/0.log" Jul 10 19:45:52.384969 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:52.384931 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-697457d895-hj6vs_d275207d-476a-4ce4-9ce2-da76a36e658d/kube-auth-proxy/0.log" Jul 10 19:45:52.614396 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:52.614355 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8c8d8cb74-pbqx5_348b64e9-4db6-4d74-9a29-36436b02e1d3/router/0.log" Jul 10 19:45:53.377992 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:53.377960 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-szfx9_3a2ad480-1b70-460f-8fe5-e0dafed541e5/serve-healthcheck-canary/0.log" Jul 10 19:45:54.017020 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:54.016987 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5fqvm_89e10ee7-fb7a-4cc2-9638-030cf5c0a305/kube-rbac-proxy/0.log" Jul 10 19:45:54.041596 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:54.041567 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5fqvm_89e10ee7-fb7a-4cc2-9638-030cf5c0a305/exporter/0.log" Jul 10 19:45:54.064341 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:54.064308 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5fqvm_89e10ee7-fb7a-4cc2-9638-030cf5c0a305/extractor/0.log" Jul 10 19:45:54.429859 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:54.429784 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-q9l4b/perf-node-gather-daemonset-cpc25" Jul 10 19:45:56.481069 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:56.480971 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-b4f45_112b1d22-d1b8-4de9-9baf-2821f4566806/manager/0.log" Jul 10 19:45:56.729978 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:56.729940 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-kkqx7_57d9470d-fca4-4f43-907d-3b40595570fb/manager/0.log" Jul 10 19:45:56.904155 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:56.904122 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-77d7f68d48-hnrxv_2e1f0282-7303-43da-ae3c-767a14ab8c16/manager/0.log" Jul 10 19:45:56.928328 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:56.928298 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-92hm5_249c9d5b-dc25-47cb-b492-cb0ca0805ea9/server/0.log" Jul 10 19:45:57.046086 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:57.046059 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-9g4cr_29957d41-1783-4280-bee2-75c4529ab426/manager/0.log" Jul 10 19:45:57.259526 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:57.259478 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7cfcf59dd-wwbcb_3c64a97c-e4f5-49d7-ad20-18a4ec847644/manager/0.log" Jul 10 19:45:57.282608 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:57.282587 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7cfcf59dd-wwbcb_3c64a97c-e4f5-49d7-ad20-18a4ec847644/copy-manifests/0.log" Jul 10 19:45:57.846570 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:57.846544 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-697457d895-hj6vs_d275207d-476a-4ce4-9ce2-da76a36e658d/kube-auth-proxy/0.log" Jul 10 19:45:58.032756 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:58.032715 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8c8d8cb74-pbqx5_348b64e9-4db6-4d74-9a29-36436b02e1d3/router/0.log" Jul 10 19:45:59.486884 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:59.486857 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-697457d895-hj6vs_d275207d-476a-4ce4-9ce2-da76a36e658d/kube-auth-proxy/0.log" Jul 10 19:45:59.659595 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:45:59.659561 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8c8d8cb74-pbqx5_348b64e9-4db6-4d74-9a29-36436b02e1d3/router/0.log" Jul 10 19:46:03.476551 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:46:03.476451 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-dkp8t_e7b8e06d-2dd5-4468-b6f8-4d62c2f6aac0/migrator/0.log" Jul 10 19:46:03.502167 ip-10-0-135-67 kubenswrapper[2581]: I0710 19:46:03.502134 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-dkp8t_e7b8e06d-2dd5-4468-b6f8-4d62c2f6aac0/graceful-termination/0.log"