Jul 10 02:07:48.303976 ip-10-0-130-65 systemd[1]: Starting Kubernetes Kubelet... Jul 10 02:07:48.854091 ip-10-0-130-65 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 02:07:48.854091 ip-10-0-130-65 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 10 02:07:48.854091 ip-10-0-130-65 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 02:07:48.854091 ip-10-0-130-65 kubenswrapper[2566]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 10 02:07:48.854091 ip-10-0-130-65 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 02:07:48.854091 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.701232 2566 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 10 02:07:48.854091 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706531 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 02:07:48.854091 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706550 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 10 02:07:48.854091 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706555 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 02:07:48.854091 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706562 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 02:07:48.854091 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706567 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 02:07:48.854091 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706570 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 02:07:48.854091 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706574 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 02:07:48.854091 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706577 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 02:07:48.784235 ip-10-0-130-65 systemd[1]: Started Kubernetes Kubelet. Jul 10 02:07:48.855489 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706581 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 02:07:48.855489 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706584 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 02:07:48.855489 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706588 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 02:07:48.855489 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706591 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 02:07:48.855489 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706593 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 02:07:48.855489 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706597 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 02:07:48.855489 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706600 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 02:07:48.855489 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706603 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 02:07:48.855489 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706606 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 02:07:48.855489 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706619 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 02:07:48.855489 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706622 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 02:07:48.855489 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706625 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 02:07:48.855489 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706629 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 02:07:48.855489 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706632 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 02:07:48.855489 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706635 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 02:07:48.855489 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706638 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 02:07:48.855489 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706641 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 02:07:48.855489 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706644 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 02:07:48.855489 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706647 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 02:07:48.855489 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706649 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 02:07:49.183299 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706652 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 02:07:49.183299 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706655 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 02:07:49.183299 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706657 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 02:07:49.183299 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706660 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 02:07:49.183299 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706665 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 02:07:49.183299 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706668 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 02:07:49.183299 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706688 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 02:07:49.183299 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706691 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 02:07:49.183299 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706694 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 02:07:49.183299 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706698 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 02:07:49.183299 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706701 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 02:07:49.183299 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706704 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 02:07:49.183299 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706707 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 02:07:49.183299 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706710 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 02:07:49.183299 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706713 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 02:07:49.183299 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706715 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 02:07:49.183299 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706718 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 02:07:49.183299 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706722 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 02:07:49.183299 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706725 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 02:07:49.196111 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706728 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 02:07:49.196111 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706731 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 02:07:49.196111 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706735 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 02:07:49.196111 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706744 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 02:07:49.196111 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706748 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 02:07:49.196111 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706751 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 02:07:49.196111 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706753 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 02:07:49.196111 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706756 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 02:07:49.196111 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706759 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 02:07:49.196111 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706763 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 02:07:49.196111 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706766 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 02:07:49.196111 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706770 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 02:07:49.196111 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706772 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 02:07:49.196111 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706775 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 02:07:49.196111 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706778 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 02:07:49.196111 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706781 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 02:07:49.196111 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706783 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 02:07:49.196111 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706786 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 02:07:49.196111 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706789 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 02:07:49.196111 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706792 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 02:07:49.196810 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706801 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 02:07:49.196810 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706804 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 02:07:49.196810 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706808 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 02:07:49.196810 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706811 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 02:07:49.196810 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706814 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 02:07:49.196810 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706817 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 02:07:49.196810 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706820 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 02:07:49.196810 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706823 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 02:07:49.196810 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706826 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 02:07:49.196810 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706828 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 02:07:49.196810 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706831 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 02:07:49.196810 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706834 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 02:07:49.196810 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706837 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 02:07:49.196810 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706840 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 02:07:49.196810 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706843 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 02:07:49.196810 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706845 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 02:07:49.196810 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706854 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 02:07:49.196810 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706857 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 02:07:49.196810 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706860 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 02:07:49.196810 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706864 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 02:07:49.197495 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706868 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 02:07:49.197495 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706871 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 02:07:49.197495 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706875 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 02:07:49.197495 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706878 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 02:07:49.197495 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706881 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 02:07:49.197495 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706884 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 02:07:49.197495 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706888 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 02:07:49.197495 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706891 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 02:07:49.197495 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706894 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 02:07:49.197495 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706896 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 02:07:49.197495 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706899 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 02:07:49.197495 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.706902 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 02:07:49.197495 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707772 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 02:07:49.197495 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707781 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 02:07:49.197495 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707786 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 02:07:49.197495 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707791 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 02:07:49.197495 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707795 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 02:07:49.197495 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707798 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 02:07:49.197495 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707801 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 02:07:49.198323 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707804 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 02:07:49.198323 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707807 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 02:07:49.198323 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707810 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 02:07:49.198323 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707813 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 02:07:49.198323 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707816 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 02:07:49.198323 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707819 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 02:07:49.198323 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707822 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 02:07:49.198323 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707825 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 02:07:49.198323 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707828 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 02:07:49.198323 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707831 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 02:07:49.198323 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707835 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 02:07:49.198323 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707838 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 02:07:49.198323 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707841 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 02:07:49.198323 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707845 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 02:07:49.198323 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707848 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 02:07:49.198323 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707851 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 02:07:49.198323 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707854 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 02:07:49.198323 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707857 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 02:07:49.198323 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707860 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 02:07:49.199181 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707862 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 02:07:49.199181 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707866 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 02:07:49.199181 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707869 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 02:07:49.199181 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707872 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 02:07:49.199181 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707875 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 02:07:49.199181 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707877 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 02:07:49.199181 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707881 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 02:07:49.199181 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707884 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 02:07:49.199181 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707887 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 02:07:49.199181 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707889 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 02:07:49.199181 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707892 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 02:07:49.199181 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707895 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 02:07:49.199181 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707898 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 02:07:49.199181 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707900 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 02:07:49.199181 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707904 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 02:07:49.199181 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707907 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 02:07:49.199181 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707910 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 02:07:49.199181 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707912 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 02:07:49.199181 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707915 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 02:07:49.199181 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707918 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 02:07:49.199823 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707921 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 02:07:49.199823 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707924 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 02:07:49.199823 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707927 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 02:07:49.199823 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707929 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 02:07:49.199823 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707933 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 02:07:49.199823 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707936 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 02:07:49.199823 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707939 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 02:07:49.199823 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707941 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 02:07:49.199823 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707944 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 02:07:49.199823 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707947 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 02:07:49.199823 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707949 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 02:07:49.199823 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707954 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 02:07:49.199823 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707957 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 02:07:49.199823 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707960 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 02:07:49.199823 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707963 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 02:07:49.199823 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707966 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 02:07:49.199823 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707968 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 02:07:49.199823 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707971 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 02:07:49.199823 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707974 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 02:07:49.200526 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707978 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 02:07:49.200526 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707980 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 02:07:49.200526 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707983 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 02:07:49.200526 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707986 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 02:07:49.200526 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707988 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 02:07:49.200526 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707992 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 02:07:49.200526 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707995 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 02:07:49.200526 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.707998 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 02:07:49.200526 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708003 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 02:07:49.200526 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708006 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 02:07:49.200526 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708009 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 02:07:49.200526 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708011 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 02:07:49.200526 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708014 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 02:07:49.200526 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708017 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 02:07:49.200526 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708020 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 02:07:49.200526 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708023 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 02:07:49.200526 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708025 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 02:07:49.200526 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708028 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 02:07:49.200526 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708031 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 02:07:49.200526 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708034 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 02:07:49.201293 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708037 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 02:07:49.201293 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708040 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 02:07:49.201293 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708042 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 02:07:49.201293 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708045 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 02:07:49.201293 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708048 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 02:07:49.201293 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708050 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 02:07:49.201293 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708053 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 02:07:49.201293 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708056 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 02:07:49.201293 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708059 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 10 02:07:49.201293 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708061 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 02:07:49.201293 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708064 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 02:07:49.201293 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708067 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 02:07:49.201293 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708069 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 02:07:49.201293 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708072 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 02:07:49.201293 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708522 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 02:07:49.201293 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708531 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 02:07:49.201293 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708534 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 02:07:49.201293 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708538 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 02:07:49.201293 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708540 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 02:07:49.201293 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708544 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 02:07:49.202130 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708547 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 02:07:49.202130 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708550 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 02:07:49.202130 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708553 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 02:07:49.202130 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708556 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 02:07:49.202130 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708559 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 02:07:49.202130 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708562 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 02:07:49.202130 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708565 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 02:07:49.202130 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708568 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 02:07:49.202130 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708571 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 02:07:49.202130 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708573 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 02:07:49.202130 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708577 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 02:07:49.202130 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708580 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 02:07:49.202130 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708583 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 02:07:49.202130 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708585 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 02:07:49.202130 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708588 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 02:07:49.202130 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708591 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 02:07:49.202130 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708595 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 02:07:49.202130 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708598 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 02:07:49.202130 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708601 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 02:07:49.202130 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708603 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 02:07:49.202817 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708606 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 10 02:07:49.202817 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708609 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 02:07:49.202817 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708611 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 02:07:49.202817 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708615 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 02:07:49.202817 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708618 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 02:07:49.202817 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708620 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 02:07:49.202817 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708623 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 02:07:49.202817 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708627 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 02:07:49.202817 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708630 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 02:07:49.202817 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708633 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 02:07:49.202817 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708636 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 02:07:49.202817 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708638 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 02:07:49.202817 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708642 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 02:07:49.202817 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708644 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 02:07:49.202817 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708648 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 02:07:49.202817 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708650 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 02:07:49.202817 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708653 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 02:07:49.202817 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708659 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 02:07:49.202817 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708661 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 02:07:49.202817 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708664 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 02:07:49.203479 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708667 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 02:07:49.203479 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708684 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 02:07:49.203479 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708687 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 02:07:49.203479 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708690 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 02:07:49.203479 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708694 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 02:07:49.203479 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708696 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 02:07:49.203479 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708701 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 02:07:49.203479 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708705 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 02:07:49.203479 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708710 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 02:07:49.203479 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708713 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 02:07:49.203479 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708717 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 02:07:49.203479 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708721 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 02:07:49.203479 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708725 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 02:07:49.203479 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708728 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 02:07:49.203479 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708731 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 02:07:49.203479 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708734 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 02:07:49.203479 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708737 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 02:07:49.203479 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708739 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 02:07:49.203479 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708742 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 02:07:49.203479 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708746 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 02:07:49.204125 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708749 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 02:07:49.204125 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708752 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 02:07:49.204125 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708755 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 02:07:49.204125 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708758 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 02:07:49.204125 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708761 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 02:07:49.204125 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708764 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 02:07:49.204125 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708766 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 02:07:49.204125 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708769 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 02:07:49.204125 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708772 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 02:07:49.204125 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708775 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 02:07:49.204125 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708777 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 02:07:49.204125 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708780 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 02:07:49.204125 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708783 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 02:07:49.204125 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708785 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 02:07:49.204125 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708788 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 02:07:49.204125 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708791 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 02:07:49.204125 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708794 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 02:07:49.204125 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708797 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 02:07:49.204125 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708799 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 02:07:49.204125 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708802 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 02:07:49.204812 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708806 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 02:07:49.204812 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708812 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 02:07:49.204812 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708815 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 02:07:49.204812 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708817 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 02:07:49.204812 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708820 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 02:07:49.204812 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708823 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 02:07:49.204812 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708826 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 02:07:49.204812 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708828 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 02:07:49.204812 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708831 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 02:07:49.204812 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708834 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 02:07:49.204812 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708837 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 02:07:49.204812 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708841 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 02:07:49.204812 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.708844 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 02:07:49.204812 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.708923 2566 flags.go:64] FLAG: --address="0.0.0.0" Jul 10 02:07:49.204812 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.708931 2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 10 02:07:49.204812 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.708938 2566 flags.go:64] FLAG: --anonymous-auth="true" Jul 10 02:07:49.204812 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.708943 2566 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 10 02:07:49.204812 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.708948 2566 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 10 02:07:49.204812 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.708952 2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 10 02:07:49.204812 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.708956 2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 10 02:07:49.205397 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.708961 2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 10 02:07:49.205397 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.708964 2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 10 02:07:49.205397 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.708968 2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 10 02:07:49.205397 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.708972 2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 10 02:07:49.205397 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.708975 2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 10 02:07:49.205397 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.708978 2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 10 02:07:49.205397 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.708981 2566 flags.go:64] FLAG: --cgroup-root="" Jul 10 02:07:49.205397 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.708985 2566 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 10 02:07:49.205397 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.708988 2566 flags.go:64] FLAG: --client-ca-file="" Jul 10 02:07:49.205397 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.708991 2566 flags.go:64] FLAG: --cloud-provider="external" Jul 10 02:07:49.205397 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.708995 2566 flags.go:64] FLAG: --cluster-dns="[]" Jul 10 02:07:49.205397 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.708999 2566 flags.go:64] FLAG: --cluster-domain="" Jul 10 02:07:49.205397 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709003 2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 10 02:07:49.205397 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709007 2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 10 02:07:49.205397 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709010 2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 10 02:07:49.205397 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709013 2566 flags.go:64] FLAG: --container-log-max-files="5" Jul 10 02:07:49.205397 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709017 2566 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 10 02:07:49.205397 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709021 2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 10 02:07:49.205397 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709024 2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 10 02:07:49.205397 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709027 2566 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 10 02:07:49.205397 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709030 2566 flags.go:64] FLAG: --contention-profiling="false" Jul 10 02:07:49.205397 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709033 2566 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 10 02:07:49.205397 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709036 2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 10 02:07:49.205397 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709041 2566 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 10 02:07:49.206138 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709044 2566 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 10 02:07:49.206138 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709049 2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 10 02:07:49.206138 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709052 2566 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 10 02:07:49.206138 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709056 2566 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 10 02:07:49.206138 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709060 2566 flags.go:64] FLAG: --enable-load-reader="false" Jul 10 02:07:49.206138 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709063 2566 flags.go:64] FLAG: --enable-server="true" Jul 10 02:07:49.206138 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709066 2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 10 02:07:49.206138 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709071 2566 flags.go:64] FLAG: --event-burst="100" Jul 10 02:07:49.206138 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709075 2566 flags.go:64] FLAG: --event-qps="50" Jul 10 02:07:49.206138 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709078 2566 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 10 02:07:49.206138 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709081 2566 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 10 02:07:49.206138 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709084 2566 flags.go:64] FLAG: --eviction-hard="" Jul 10 02:07:49.206138 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709088 2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 10 02:07:49.206138 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709091 2566 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 10 02:07:49.206138 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709095 2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 10 02:07:49.206138 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709098 2566 flags.go:64] FLAG: --eviction-soft="" Jul 10 02:07:49.206138 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709101 2566 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 10 02:07:49.206138 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709104 2566 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 10 02:07:49.206138 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709108 2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 10 02:07:49.206138 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709111 2566 flags.go:64] FLAG: --experimental-mounter-path="" Jul 10 02:07:49.206138 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709116 2566 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 10 02:07:49.206138 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709119 2566 flags.go:64] FLAG: --fail-swap-on="true" Jul 10 02:07:49.206138 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709122 2566 flags.go:64] FLAG: --feature-gates="" Jul 10 02:07:49.206138 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709126 2566 flags.go:64] FLAG: --file-check-frequency="20s" Jul 10 02:07:49.206138 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709129 2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 10 02:07:49.206899 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709133 2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 10 02:07:49.206899 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709136 2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 10 02:07:49.206899 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709139 2566 flags.go:64] FLAG: --healthz-port="10248" Jul 10 02:07:49.206899 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709142 2566 flags.go:64] FLAG: --help="false" Jul 10 02:07:49.206899 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709145 2566 flags.go:64] FLAG: --hostname-override="ip-10-0-130-65.ec2.internal" Jul 10 02:07:49.206899 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709148 2566 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 10 02:07:49.206899 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709156 2566 flags.go:64] FLAG: --http-check-frequency="20s" Jul 10 02:07:49.206899 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709159 2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 10 02:07:49.206899 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709163 2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 10 02:07:49.206899 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709166 2566 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 10 02:07:49.206899 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709169 2566 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 10 02:07:49.206899 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709172 2566 flags.go:64] FLAG: --image-service-endpoint="" Jul 10 02:07:49.206899 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709175 2566 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 10 02:07:49.206899 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709178 2566 flags.go:64] FLAG: --kube-api-burst="100" Jul 10 02:07:49.206899 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709181 2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 10 02:07:49.206899 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709185 2566 flags.go:64] FLAG: --kube-api-qps="50" Jul 10 02:07:49.206899 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709187 2566 flags.go:64] FLAG: --kube-reserved="" Jul 10 02:07:49.206899 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709191 2566 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 10 02:07:49.206899 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709194 2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 10 02:07:49.206899 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709197 2566 flags.go:64] FLAG: --kubelet-cgroups="" Jul 10 02:07:49.206899 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709200 2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 10 02:07:49.206899 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709203 2566 flags.go:64] FLAG: --lock-file="" Jul 10 02:07:49.206899 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709206 2566 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 10 02:07:49.206899 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709209 2566 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 10 02:07:49.207631 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709212 2566 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 10 02:07:49.207631 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709218 2566 flags.go:64] FLAG: --log-json-split-stream="false" Jul 10 02:07:49.207631 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709222 2566 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 10 02:07:49.207631 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709227 2566 flags.go:64] FLAG: --log-text-split-stream="false" Jul 10 02:07:49.207631 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709230 2566 flags.go:64] FLAG: --logging-format="text" Jul 10 02:07:49.207631 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709233 2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 10 02:07:49.207631 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709236 2566 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 10 02:07:49.207631 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709239 2566 flags.go:64] FLAG: --manifest-url="" Jul 10 02:07:49.207631 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709242 2566 flags.go:64] FLAG: --manifest-url-header="" Jul 10 02:07:49.207631 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709247 2566 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 10 02:07:49.207631 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709250 2566 flags.go:64] FLAG: --max-open-files="1000000" Jul 10 02:07:49.207631 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709255 2566 flags.go:64] FLAG: --max-pods="110" Jul 10 02:07:49.207631 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709258 2566 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 10 02:07:49.207631 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709261 2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 10 02:07:49.207631 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709265 2566 flags.go:64] FLAG: --memory-manager-policy="None" Jul 10 02:07:49.207631 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709269 2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 10 02:07:49.207631 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709272 2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 10 02:07:49.207631 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709275 2566 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 10 02:07:49.207631 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709278 2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 10 02:07:49.207631 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709285 2566 flags.go:64] FLAG: --node-status-max-images="50" Jul 10 02:07:49.207631 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709288 2566 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 10 02:07:49.207631 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709292 2566 flags.go:64] FLAG: --oom-score-adj="-999" Jul 10 02:07:49.207631 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709296 2566 flags.go:64] FLAG: --pod-cidr="" Jul 10 02:07:49.208322 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709299 2566 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 10 02:07:49.208322 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709304 2566 flags.go:64] FLAG: --pod-manifest-path="" Jul 10 02:07:49.208322 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709308 2566 flags.go:64] FLAG: --pod-max-pids="-1" Jul 10 02:07:49.208322 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709311 2566 flags.go:64] FLAG: --pods-per-core="0" Jul 10 02:07:49.208322 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709314 2566 flags.go:64] FLAG: --port="10250" Jul 10 02:07:49.208322 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709317 2566 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 10 02:07:49.208322 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709320 2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-099c5e48af72d086d" Jul 10 02:07:49.208322 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709324 2566 flags.go:64] FLAG: --qos-reserved="" Jul 10 02:07:49.208322 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709327 2566 flags.go:64] FLAG: --read-only-port="10255" Jul 10 02:07:49.208322 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709330 2566 flags.go:64] FLAG: --register-node="true" Jul 10 02:07:49.208322 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709333 2566 flags.go:64] FLAG: --register-with-taints="" Jul 10 02:07:49.208322 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709337 2566 flags.go:64] FLAG: --registry-burst="10" Jul 10 02:07:49.208322 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709341 2566 flags.go:64] FLAG: --registry-qps="5" Jul 10 02:07:49.208322 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709346 2566 flags.go:64] FLAG: --reserved-cpus="" Jul 10 02:07:49.208322 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709349 2566 flags.go:64] FLAG: --reserved-memory="" Jul 10 02:07:49.208322 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709353 2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 10 02:07:49.208322 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709356 2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 10 02:07:49.208322 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709359 2566 flags.go:64] FLAG: --rotate-certificates="false" Jul 10 02:07:49.208322 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709363 2566 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 10 02:07:49.208322 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709366 2566 flags.go:64] FLAG: --runonce="false" Jul 10 02:07:49.208322 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709369 2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 10 02:07:49.208322 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709372 2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 10 02:07:49.208322 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709375 2566 flags.go:64] FLAG: --seccomp-default="false" Jul 10 02:07:49.208322 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709379 2566 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 10 02:07:49.208322 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709382 2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 10 02:07:49.209188 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709386 2566 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 10 02:07:49.209188 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709389 2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 10 02:07:49.209188 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709392 2566 flags.go:64] FLAG: --storage-driver-password="root" Jul 10 02:07:49.209188 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709395 2566 flags.go:64] FLAG: --storage-driver-secure="false" Jul 10 02:07:49.209188 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709398 2566 flags.go:64] FLAG: --storage-driver-table="stats" Jul 10 02:07:49.209188 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709401 2566 flags.go:64] FLAG: --storage-driver-user="root" Jul 10 02:07:49.209188 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709404 2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 10 02:07:49.209188 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709407 2566 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 10 02:07:49.209188 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709410 2566 flags.go:64] FLAG: --system-cgroups="" Jul 10 02:07:49.209188 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709413 2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 10 02:07:49.209188 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709419 2566 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 10 02:07:49.209188 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709422 2566 flags.go:64] FLAG: --tls-cert-file="" Jul 10 02:07:49.209188 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709426 2566 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 10 02:07:49.209188 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709431 2566 flags.go:64] FLAG: --tls-min-version="" Jul 10 02:07:49.209188 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709434 2566 flags.go:64] FLAG: --tls-private-key-file="" Jul 10 02:07:49.209188 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709436 2566 flags.go:64] FLAG: --topology-manager-policy="none" Jul 10 02:07:49.209188 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709440 2566 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 10 02:07:49.209188 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709443 2566 flags.go:64] FLAG: --topology-manager-scope="container" Jul 10 02:07:49.209188 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709446 2566 flags.go:64] FLAG: --v="2" Jul 10 02:07:49.209188 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709450 2566 flags.go:64] FLAG: --version="false" Jul 10 02:07:49.209188 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709457 2566 flags.go:64] FLAG: --vmodule="" Jul 10 02:07:49.209188 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709462 2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 10 02:07:49.209188 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.709465 2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 10 02:07:49.209188 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709588 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 02:07:49.209188 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709592 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 02:07:49.210238 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709595 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 02:07:49.210238 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709598 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 02:07:49.210238 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709601 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 02:07:49.210238 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709604 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 02:07:49.210238 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709607 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 02:07:49.210238 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709611 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 02:07:49.210238 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709614 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 02:07:49.210238 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709617 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 02:07:49.210238 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709619 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 02:07:49.210238 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709622 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 02:07:49.210238 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709625 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 02:07:49.210238 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709628 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 02:07:49.210238 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709631 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 02:07:49.210238 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709634 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 02:07:49.210238 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709637 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 02:07:49.210238 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709640 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 02:07:49.210238 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709642 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 02:07:49.210238 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709645 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 02:07:49.210238 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709648 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 02:07:49.210238 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709651 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 02:07:49.211115 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709653 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 02:07:49.211115 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709656 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 02:07:49.211115 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709659 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 02:07:49.211115 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709662 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 02:07:49.211115 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709665 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 02:07:49.211115 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709667 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 02:07:49.211115 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709684 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 02:07:49.211115 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709689 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 02:07:49.211115 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709693 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 02:07:49.211115 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709696 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 02:07:49.211115 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709699 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 02:07:49.211115 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709701 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 02:07:49.211115 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709704 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 02:07:49.211115 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709707 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 02:07:49.211115 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709710 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 02:07:49.211115 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709713 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 02:07:49.211115 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709716 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 02:07:49.211115 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709720 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 02:07:49.211115 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709723 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 02:07:49.211115 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709726 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 02:07:49.481635 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709729 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 02:07:49.481635 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709732 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 02:07:49.481635 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709736 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 02:07:49.481635 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709740 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 02:07:49.481635 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709743 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 02:07:49.481635 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709746 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 02:07:49.481635 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709749 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 02:07:49.481635 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709752 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 02:07:49.481635 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709755 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 02:07:49.481635 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709758 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 02:07:49.481635 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709761 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 02:07:49.481635 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709764 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 02:07:49.481635 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709766 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 02:07:49.481635 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709769 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 02:07:49.481635 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709772 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 10 02:07:49.481635 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709775 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 02:07:49.481635 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709778 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 02:07:49.481635 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709780 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 02:07:49.481635 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709783 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 02:07:49.483071 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709787 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 02:07:49.483071 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709790 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 02:07:49.483071 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709794 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 02:07:49.483071 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709797 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 02:07:49.483071 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709799 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 02:07:49.483071 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709802 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 02:07:49.483071 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709805 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 02:07:49.483071 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709807 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 02:07:49.483071 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709812 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 02:07:49.483071 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709816 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 02:07:49.483071 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709821 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 02:07:49.483071 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709824 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 02:07:49.483071 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709827 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 02:07:49.483071 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709830 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 02:07:49.483071 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709834 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 02:07:49.483071 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709838 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 02:07:49.483071 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709841 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 02:07:49.483071 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709844 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 02:07:49.483071 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709846 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 02:07:49.483698 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709849 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 02:07:49.483698 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709852 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 02:07:49.483698 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709855 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 02:07:49.483698 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709858 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 02:07:49.483698 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709861 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 02:07:49.483698 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709864 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 02:07:49.483698 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709866 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 02:07:49.483698 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709869 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 02:07:49.483698 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709872 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 02:07:49.483698 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709875 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 02:07:49.483698 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709877 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 02:07:49.483698 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709880 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 02:07:49.483698 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709883 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 02:07:49.483698 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709887 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 02:07:49.483698 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709890 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 02:07:49.483698 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709893 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 02:07:49.483698 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709896 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 02:07:49.483698 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709899 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 02:07:49.483698 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.709902 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 02:07:49.484320 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.710612 2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 02:07:49.484320 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.718063 2566 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 10 02:07:49.484320 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.718084 2566 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 10 02:07:49.484320 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.718106 2566 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 10 02:07:49.484320 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.718113 2566 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 10 02:07:49.484320 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718179 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 02:07:49.484320 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718184 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 02:07:49.484320 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718188 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 02:07:49.484320 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718192 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 02:07:49.484320 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718195 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 02:07:49.484320 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718198 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 02:07:49.484320 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718201 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 02:07:49.484320 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718204 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 02:07:49.484320 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718207 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 02:07:49.484320 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718210 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 02:07:49.484837 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718212 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 02:07:49.484837 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718215 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 02:07:49.484837 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718221 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 02:07:49.484837 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718225 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 02:07:49.484837 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718228 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 02:07:49.484837 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718232 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 02:07:49.484837 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718235 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 02:07:49.484837 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718241 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 02:07:49.484837 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718244 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 02:07:49.484837 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718247 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 02:07:49.484837 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718249 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 02:07:49.484837 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718253 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 02:07:49.484837 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718255 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 02:07:49.484837 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718259 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 02:07:49.484837 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718261 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 02:07:49.484837 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718264 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 02:07:49.484837 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718267 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 02:07:49.484837 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718270 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 02:07:49.484837 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718273 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 02:07:49.484837 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718277 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 02:07:49.485482 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718280 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 02:07:49.485482 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718285 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 02:07:49.485482 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718288 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 02:07:49.485482 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718291 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 02:07:49.485482 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718294 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 02:07:49.485482 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718297 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 02:07:49.485482 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718301 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 02:07:49.485482 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718304 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 02:07:49.485482 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718307 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 02:07:49.485482 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718310 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 02:07:49.485482 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718313 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 02:07:49.485482 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718315 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 02:07:49.485482 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718318 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 02:07:49.485482 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718321 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 02:07:49.485482 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718324 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 02:07:49.485482 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718327 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 02:07:49.485482 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718330 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 02:07:49.485482 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718332 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 02:07:49.485482 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718335 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 02:07:49.485482 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718341 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 02:07:49.486101 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718344 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 10 02:07:49.486101 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718347 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 02:07:49.486101 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718349 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 02:07:49.486101 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718352 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 02:07:49.486101 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718355 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 02:07:49.486101 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718358 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 02:07:49.486101 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718361 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 02:07:49.486101 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718363 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 02:07:49.486101 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718366 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 02:07:49.486101 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718370 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 02:07:49.486101 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718373 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 02:07:49.486101 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718377 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 02:07:49.486101 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718381 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 02:07:49.486101 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718386 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 02:07:49.486101 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718390 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 02:07:49.486101 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718393 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 02:07:49.486101 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718396 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 02:07:49.486101 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718399 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 02:07:49.486101 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718402 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 02:07:49.486761 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718405 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 02:07:49.486761 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718408 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 02:07:49.486761 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718410 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 02:07:49.486761 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718413 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 02:07:49.486761 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718416 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 02:07:49.486761 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718418 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 02:07:49.486761 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718421 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 02:07:49.486761 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718424 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 02:07:49.486761 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718427 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 02:07:49.486761 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718429 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 02:07:49.486761 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718432 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 02:07:49.486761 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718435 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 02:07:49.486761 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718439 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 02:07:49.486761 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718441 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 02:07:49.486761 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718444 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 02:07:49.486761 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718447 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 02:07:49.486761 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718450 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 02:07:49.486761 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718452 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 02:07:49.486761 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718455 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 02:07:49.487330 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718458 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 02:07:49.487330 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718461 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 02:07:49.487330 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718463 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 02:07:49.487330 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718466 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 02:07:49.487330 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718469 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 02:07:49.487330 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718471 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 02:07:49.487330 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718474 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 02:07:49.487330 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718479 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 02:07:49.487330 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718482 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 02:07:49.487330 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718485 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 02:07:49.487330 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718487 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 02:07:49.487330 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.718493 2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 02:07:49.487330 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718637 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 02:07:49.487330 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718641 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 02:07:49.487330 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718645 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 02:07:49.487845 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718648 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 02:07:49.487845 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718651 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 02:07:49.487845 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718654 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 02:07:49.487845 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718657 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 02:07:49.487845 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718660 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 02:07:49.487845 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718663 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 02:07:49.487845 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718666 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 02:07:49.487845 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718669 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 02:07:49.487845 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718692 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 02:07:49.487845 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718695 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 02:07:49.487845 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718700 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 02:07:49.487845 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718703 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 02:07:49.487845 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718706 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 02:07:49.487845 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718710 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 02:07:49.487845 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718713 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 02:07:49.487845 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718716 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 02:07:49.487845 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718718 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 02:07:49.487845 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718721 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 02:07:49.487845 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718724 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 02:07:49.488475 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718727 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 02:07:49.488475 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718730 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 02:07:49.488475 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718732 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 02:07:49.488475 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718735 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 02:07:49.488475 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718738 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 02:07:49.488475 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718741 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 02:07:49.488475 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718748 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 02:07:49.488475 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718752 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 02:07:49.488475 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718754 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 02:07:49.488475 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718759 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 02:07:49.488475 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718762 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 02:07:49.488475 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718766 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 02:07:49.488475 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718769 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 02:07:49.488475 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718772 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 02:07:49.488475 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718775 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 02:07:49.488475 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718778 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 02:07:49.488475 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718781 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 02:07:49.488475 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718784 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 02:07:49.488475 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718787 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 02:07:49.489193 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718790 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 02:07:49.489193 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718793 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 02:07:49.489193 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718795 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 02:07:49.489193 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718798 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 02:07:49.489193 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718802 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 02:07:49.489193 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718805 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 10 02:07:49.489193 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718808 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 02:07:49.489193 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718811 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 02:07:49.489193 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718814 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 02:07:49.489193 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718816 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 02:07:49.489193 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718819 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 02:07:49.489193 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718821 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 02:07:49.489193 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718824 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 02:07:49.489193 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718827 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 02:07:49.489193 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718830 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 02:07:49.489193 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718832 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 02:07:49.489193 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718835 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 02:07:49.489193 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718838 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 02:07:49.489193 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718841 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 02:07:49.489193 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718845 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 02:07:49.489193 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718848 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 02:07:49.489909 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718851 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 02:07:49.489909 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718854 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 02:07:49.489909 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718857 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 02:07:49.489909 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718860 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 02:07:49.489909 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718863 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 02:07:49.489909 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718865 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 02:07:49.489909 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718868 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 02:07:49.489909 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718871 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 02:07:49.489909 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718874 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 02:07:49.489909 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718877 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 02:07:49.489909 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718879 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 02:07:49.489909 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718882 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 02:07:49.489909 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718885 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 02:07:49.489909 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718888 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 02:07:49.489909 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718892 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 02:07:49.489909 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718897 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 02:07:49.489909 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718899 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 02:07:49.489909 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718903 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 02:07:49.489909 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718906 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 02:07:49.490615 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718909 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 02:07:49.490615 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718912 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 02:07:49.490615 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718915 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 02:07:49.490615 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718918 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 02:07:49.490615 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718921 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 02:07:49.490615 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718924 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 02:07:49.490615 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718928 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 02:07:49.490615 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718930 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 02:07:49.490615 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718933 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 02:07:49.490615 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718936 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 02:07:49.490615 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718939 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 02:07:49.490615 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718943 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 02:07:49.490615 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718946 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 02:07:49.490615 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718949 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 02:07:49.490615 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718952 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 02:07:49.490615 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718955 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 02:07:49.490615 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718958 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 02:07:49.490615 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:48.718961 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 02:07:49.491341 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.718966 2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 02:07:49.491341 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.719091 2566 server.go:962] "Client rotation is on, will bootstrap in background" Jul 10 02:07:49.491341 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.722394 2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 10 02:07:49.491341 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.723843 2566 server.go:1019] "Starting client certificate rotation" Jul 10 02:07:49.491341 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.724123 2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 02:07:49.491341 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.724887 2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 02:07:49.491341 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.727009 2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 02:07:49.491341 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.729012 2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 02:07:49.491341 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.740101 2566 log.go:25] "Validated CRI v1 runtime API" Jul 10 02:07:49.491341 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.744322 2566 log.go:25] "Validated CRI v1 image API" Jul 10 02:07:49.491341 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.745157 2566 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 10 02:07:49.491341 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.749318 2566 fs.go:135] Filesystem UUIDs: map[6bb362cf-81e9-4bc3-9081-3a4853062bac:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 afee09cb-183f-4787-8635-61e9c6b7867b:/dev/nvme0n1p4] Jul 10 02:07:49.491997 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.749339 2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 10 02:07:49.492202 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.756134 2566 manager.go:216] Machine: {Timestamp:2026-07-10 02:07:48.754434075 +0000 UTC m=+0.330711155 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100339 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f7ab2ec221fad379cb70d0b4d1163 SystemUUID:ec2f7ab2-ec22-1fad-379c-b70d0b4d1163 BootID:89a413f5-25ce-4deb-8600-01791c6a480b Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:fa:f4:23:ed:2b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:fa:f4:23:ed:2b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:36:03:99:a6:0c:54 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 10 02:07:49.492202 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.756263 2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 10 02:07:49.492202 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.756368 2566 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 10 02:07:49.492202 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.759699 2566 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 10 02:07:49.492202 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.759730 2566 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-65.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 10 02:07:49.492202 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.760737 2566 topology_manager.go:138] "Creating topology manager with none policy" Jul 10 02:07:49.492202 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.760752 2566 container_manager_linux.go:309] "Creating device plugin manager" Jul 10 02:07:49.492202 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.760760 2566 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 02:07:49.492638 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.761472 2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 02:07:49.492638 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.762314 2566 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 10 02:07:49.492638 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.765605 2566 state_mem.go:36] "Initialized new in-memory state store" Jul 10 02:07:49.492638 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.765762 2566 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 10 02:07:49.492638 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.766688 2566 kubelet.go:486] "Attempting to sync node with API server" Jul 10 02:07:49.492638 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.766712 2566 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 10 02:07:49.492638 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.766728 2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 10 02:07:49.492638 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.766744 2566 kubelet.go:398] "Adding apiserver pod source" Jul 10 02:07:49.492638 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.766756 2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 10 02:07:49.492638 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.767265 2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 02:07:49.492638 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.767287 2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 02:07:49.492638 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.769703 2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 02:07:49.492638 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.772047 2566 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 10 02:07:49.492638 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.777824 2566 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 10 02:07:49.492638 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.777861 2566 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 10 02:07:49.492638 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.778532 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 10 02:07:49.492638 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.778545 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 10 02:07:49.492638 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.778552 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 10 02:07:49.492638 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.778558 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 10 02:07:49.492638 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.778569 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 10 02:07:49.492638 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.778579 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 10 02:07:49.492638 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.778626 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 10 02:07:49.492638 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.778632 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 10 02:07:49.492638 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.778640 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 10 02:07:49.492638 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.778646 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 10 02:07:49.492638 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.778655 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 10 02:07:49.492638 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.778664 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 10 02:07:49.492638 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.779468 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 10 02:07:49.492638 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.779476 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 10 02:07:49.492638 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:48.779778 2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-65.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 10 02:07:49.492638 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:48.779821 2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 10 02:07:49.492638 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.782592 2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-65.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 10 02:07:49.492638 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.783315 2566 server.go:1268] "Started kubelet" Jul 10 02:07:49.492638 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.783381 2566 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 10 02:07:49.492638 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.783417 2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 10 02:07:49.492638 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.783502 2566 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 10 02:07:49.492638 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.784610 2566 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 10 02:07:49.492638 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.786432 2566 server.go:310] "Adding debug handlers to kubelet server" Jul 10 02:07:49.492638 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:48.789378 2566 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 10 02:07:49.493918 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.789662 2566 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 10 02:07:49.493918 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.789695 2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 10 02:07:49.493918 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:48.789879 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-65.ec2.internal\" not found" Jul 10 02:07:49.493918 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.790454 2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 10 02:07:49.493918 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.790514 2566 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 10 02:07:49.493918 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.790533 2566 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 10 02:07:49.493918 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:48.789662 2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-65.ec2.internal.18c0cb8f4db68902 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-65.ec2.internal,UID:ip-10-0-130-65.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-65.ec2.internal,},FirstTimestamp:2026-07-10 02:07:48.78328653 +0000 UTC m=+0.359563610,LastTimestamp:2026-07-10 02:07:48.78328653 +0000 UTC m=+0.359563610,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-65.ec2.internal,}" Jul 10 02:07:49.493918 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.791303 2566 reconstruct.go:97] "Volume reconstruction finished" Jul 10 02:07:49.493918 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.791313 2566 reconciler.go:29] "Reconciler: start to sync state" Jul 10 02:07:49.493918 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.791568 2566 factory.go:55] Registering systemd factory Jul 10 02:07:49.493918 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.791586 2566 factory.go:223] Registration of the systemd container factory successfully Jul 10 02:07:49.493918 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.794055 2566 factory.go:153] Registering CRI-O factory Jul 10 02:07:49.493918 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.794069 2566 factory.go:223] Registration of the crio container factory successfully Jul 10 02:07:49.493918 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.794118 2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 10 02:07:49.493918 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.794145 2566 factory.go:103] Registering Raw factory Jul 10 02:07:49.493918 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.794159 2566 manager.go:1196] Started watching for new ooms in manager Jul 10 02:07:49.493918 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:48.794980 2566 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-65.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 10 02:07:49.493918 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:48.795009 2566 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 10 02:07:49.493918 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.795235 2566 manager.go:349] Starting recovery of all containers Jul 10 02:07:49.493918 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.801223 2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-992gz" Jul 10 02:07:49.493918 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.808741 2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-992gz" Jul 10 02:07:49.493918 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.811702 2566 manager.go:354] Recovery completed Jul 10 02:07:49.493918 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.816490 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 02:07:49.493918 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.818638 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-65.ec2.internal" event="NodeHasSufficientMemory" Jul 10 02:07:49.493918 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.818668 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-65.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 02:07:49.493918 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.818700 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-65.ec2.internal" event="NodeHasSufficientPID" Jul 10 02:07:49.493918 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.819335 2566 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 10 02:07:49.493918 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.819358 2566 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 10 02:07:49.493918 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.819375 2566 state_mem.go:36] "Initialized new in-memory state store" Jul 10 02:07:49.493918 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.819349 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 02:07:49.493918 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.820249 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-65.ec2.internal" event="NodeHasSufficientMemory" Jul 10 02:07:49.493918 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.820276 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-65.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 02:07:49.493918 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.820286 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-65.ec2.internal" event="NodeHasSufficientPID" Jul 10 02:07:49.494922 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.820489 2566 policy_none.go:49] "None policy: Start" Jul 10 02:07:49.494922 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.820504 2566 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 10 02:07:49.494922 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.820515 2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 10 02:07:49.494922 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:48.821252 2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-65.ec2.internal.18c0cb8f4fd22d5c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-65.ec2.internal,UID:ip-10-0-130-65.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-65.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-65.ec2.internal,},FirstTimestamp:2026-07-10 02:07:48.818652508 +0000 UTC m=+0.394929587,LastTimestamp:2026-07-10 02:07:48.818652508 +0000 UTC m=+0.394929587,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-65.ec2.internal,}" Jul 10 02:07:49.494922 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.821618 2566 policy_none.go:47] "Start" Jul 10 02:07:49.494922 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.861302 2566 manager.go:337] "Starting Device Plugin manager" Jul 10 02:07:49.494922 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:48.861496 2566 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 10 02:07:49.494922 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.861511 2566 server.go:87] "Starting device plugin registration server" Jul 10 02:07:49.494922 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.861769 2566 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 10 02:07:49.494922 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.861779 2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 10 02:07:49.494922 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.861900 2566 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 10 02:07:49.494922 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.861990 2566 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 10 02:07:49.494922 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.862002 2566 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 10 02:07:49.494922 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:48.862638 2566 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 10 02:07:49.494922 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:48.862695 2566 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-65.ec2.internal\" not found" Jul 10 02:07:49.494922 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.962659 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 02:07:49.494922 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.963738 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-65.ec2.internal" event="NodeHasSufficientMemory" Jul 10 02:07:49.494922 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.963776 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-65.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 02:07:49.494922 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.963795 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-65.ec2.internal" event="NodeHasSufficientPID" Jul 10 02:07:49.494922 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.963832 2566 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-65.ec2.internal" Jul 10 02:07:49.494922 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:48.973340 2566 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-65.ec2.internal" Jul 10 02:07:49.494922 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:48.973367 2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-65.ec2.internal\": node \"ip-10-0-130-65.ec2.internal\" not found" Jul 10 02:07:49.494922 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:48.990921 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-65.ec2.internal\" not found" Jul 10 02:07:49.494922 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:49.003179 2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 10 02:07:49.494922 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:49.004508 2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 10 02:07:49.494922 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:49.004550 2566 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 10 02:07:49.494922 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:49.004573 2566 kubelet.go:2448] "Starting kubelet main sync loop" Jul 10 02:07:49.494922 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:49.004615 2566 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 10 02:07:49.494922 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:49.006469 2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 02:07:49.494922 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:49.094880 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-65.ec2.internal\" not found" Jul 10 02:07:49.494922 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:49.104703 2566 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-65.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-65.ec2.internal"] Jul 10 02:07:49.494922 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:49.104780 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 02:07:49.494922 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:49.106043 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-65.ec2.internal" event="NodeHasSufficientMemory" Jul 10 02:07:49.798599 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:49.106075 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-65.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 02:07:49.798599 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:49.106086 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-65.ec2.internal" event="NodeHasSufficientPID" Jul 10 02:07:49.798599 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:49.107533 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 02:07:49.798599 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:49.107722 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-65.ec2.internal" Jul 10 02:07:49.798599 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:49.107762 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 02:07:49.798599 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:49.108664 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-65.ec2.internal" event="NodeHasSufficientMemory" Jul 10 02:07:49.798599 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:49.108688 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-65.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 02:07:49.798599 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:49.108699 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-65.ec2.internal" event="NodeHasSufficientPID" Jul 10 02:07:49.798599 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:49.108669 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-65.ec2.internal" event="NodeHasSufficientMemory" Jul 10 02:07:49.798599 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:49.108767 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-65.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 02:07:49.798599 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:49.108783 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-65.ec2.internal" event="NodeHasSufficientPID" Jul 10 02:07:49.798599 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:49.110018 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-65.ec2.internal" Jul 10 02:07:49.798599 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:49.110047 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 02:07:49.798599 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:49.110790 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-65.ec2.internal" event="NodeHasSufficientMemory" Jul 10 02:07:49.798599 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:49.110808 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-65.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 02:07:49.798599 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:49.110832 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-65.ec2.internal" event="NodeHasSufficientPID" Jul 10 02:07:49.798599 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:49.140629 2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-65.ec2.internal\" not found" node="ip-10-0-130-65.ec2.internal" Jul 10 02:07:49.798599 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:49.145722 2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-65.ec2.internal\" not found" node="ip-10-0-130-65.ec2.internal" Jul 10 02:07:49.798599 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:49.195444 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-65.ec2.internal\" not found" Jul 10 02:07:49.798599 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:49.195532 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/aff7f4ed066dfa82229540d0126815cc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-65.ec2.internal\" (UID: \"aff7f4ed066dfa82229540d0126815cc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-65.ec2.internal" Jul 10 02:07:49.798599 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:49.195565 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aff7f4ed066dfa82229540d0126815cc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-65.ec2.internal\" (UID: \"aff7f4ed066dfa82229540d0126815cc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-65.ec2.internal" Jul 10 02:07:49.798599 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:49.195609 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/305afbb58c749f5158dc47cb0c52154f-config\") pod \"kube-apiserver-proxy-ip-10-0-130-65.ec2.internal\" (UID: \"305afbb58c749f5158dc47cb0c52154f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-65.ec2.internal" Jul 10 02:07:49.798599 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:49.296389 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-65.ec2.internal\" not found" Jul 10 02:07:49.798599 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:49.296464 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/aff7f4ed066dfa82229540d0126815cc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-65.ec2.internal\" (UID: \"aff7f4ed066dfa82229540d0126815cc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-65.ec2.internal" Jul 10 02:07:49.799447 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:49.296480 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/aff7f4ed066dfa82229540d0126815cc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-65.ec2.internal\" (UID: \"aff7f4ed066dfa82229540d0126815cc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-65.ec2.internal" Jul 10 02:07:49.799447 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:49.296548 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aff7f4ed066dfa82229540d0126815cc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-65.ec2.internal\" (UID: \"aff7f4ed066dfa82229540d0126815cc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-65.ec2.internal" Jul 10 02:07:49.799447 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:49.296571 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/305afbb58c749f5158dc47cb0c52154f-config\") pod \"kube-apiserver-proxy-ip-10-0-130-65.ec2.internal\" (UID: \"305afbb58c749f5158dc47cb0c52154f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-65.ec2.internal" Jul 10 02:07:49.799447 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:49.296596 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/305afbb58c749f5158dc47cb0c52154f-config\") pod \"kube-apiserver-proxy-ip-10-0-130-65.ec2.internal\" (UID: \"305afbb58c749f5158dc47cb0c52154f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-65.ec2.internal" Jul 10 02:07:49.799447 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:49.296635 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aff7f4ed066dfa82229540d0126815cc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-65.ec2.internal\" (UID: \"aff7f4ed066dfa82229540d0126815cc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-65.ec2.internal" Jul 10 02:07:49.799447 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:49.396912 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-65.ec2.internal\" not found" Jul 10 02:07:49.799447 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:49.445310 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-65.ec2.internal" Jul 10 02:07:49.799447 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:49.449116 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-65.ec2.internal" Jul 10 02:07:49.799447 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:49.497431 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-65.ec2.internal\" not found" Jul 10 02:07:49.799447 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:49.598225 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-65.ec2.internal\" not found" Jul 10 02:07:49.799447 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:49.699114 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-65.ec2.internal\" not found" Jul 10 02:07:49.799447 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:49.725230 2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 10 02:07:49.799447 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:49.725411 2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 02:07:49.799447 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:49.777966 2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 02:07:49.800984 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:49.799537 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-65.ec2.internal\" not found" Jul 10 02:07:50.134813 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:49.813401 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-09 02:02:49 +0000 UTC" deadline="2028-03-10 13:03:33.606584372 +0000 UTC" Jul 10 02:07:50.134813 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:49.813450 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14626h55m43.793139188s" Jul 10 02:07:50.134813 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:49.899945 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-65.ec2.internal\" not found" Jul 10 02:07:50.134813 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:49.935800 2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 02:07:50.134813 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:49.993343 2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-65.ec2.internal" Jul 10 02:07:50.134813 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.003263 2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 02:07:50.134813 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.004207 2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-65.ec2.internal" Jul 10 02:07:50.134813 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.015309 2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 02:07:50.440697 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.240249 2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 02:07:50.440697 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:50.252981 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaff7f4ed066dfa82229540d0126815cc.slice/crio-29ebf9b96422ba536ee97f44a7c633d86635e0588351d4347224cf74eadbae8d WatchSource:0}: Error finding container 29ebf9b96422ba536ee97f44a7c633d86635e0588351d4347224cf74eadbae8d: Status 404 returned error can't find the container with id 29ebf9b96422ba536ee97f44a7c633d86635e0588351d4347224cf74eadbae8d Jul 10 02:07:50.440697 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:50.253358 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod305afbb58c749f5158dc47cb0c52154f.slice/crio-09853460d34b6a91f2b87bf216f6aa73e5f81ac3a370e05da0d4466e57511872 WatchSource:0}: Error finding container 09853460d34b6a91f2b87bf216f6aa73e5f81ac3a370e05da0d4466e57511872: Status 404 returned error can't find the container with id 09853460d34b6a91f2b87bf216f6aa73e5f81ac3a370e05da0d4466e57511872 Jul 10 02:07:50.440697 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.257714 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 02:07:50.851912 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.767336 2566 apiserver.go:52] "Watching apiserver" Jul 10 02:07:50.851912 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.776651 2566 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 10 02:07:50.851912 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.777113 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-6hfgs","openshift-network-operator/iptables-alerter-58txz","openshift-ovn-kubernetes/ovnkube-node-d8d5l","kube-system/kube-apiserver-proxy-ip-10-0-130-65.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vtbg2","openshift-cluster-node-tuning-operator/tuned-68dsx","openshift-image-registry/node-ca-8t46j","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-65.ec2.internal","openshift-multus/multus-additional-cni-plugins-f9csf","openshift-multus/multus-wpxd6","openshift-network-diagnostics/network-check-target-ll898","kube-system/global-pull-secret-syncer-pcfzz","kube-system/konnectivity-agent-cprsf","openshift-dns/node-resolver-xgndm"] Jul 10 02:07:50.851912 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.791203 2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 10 02:07:50.851912 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.791523 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wpxd6" Jul 10 02:07:50.851912 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.794381 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cprsf" Jul 10 02:07:50.851912 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.794535 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 10 02:07:50.851912 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.794582 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-x95k8\"" Jul 10 02:07:50.851912 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.794982 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 10 02:07:50.851912 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.795020 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 10 02:07:50.851912 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.795087 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 10 02:07:50.851912 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.796993 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-z7w5m\"" Jul 10 02:07:50.851912 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.797303 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 10 02:07:50.851912 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.797502 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 10 02:07:50.851912 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.798013 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:50.851912 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.800486 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xgndm" Jul 10 02:07:50.851912 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.802823 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 10 02:07:50.851912 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.803003 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 10 02:07:50.851912 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.807312 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a49e0328-71c1-43ce-80e5-83d0c0862cf9-system-cni-dir\") pod \"multus-wpxd6\" (UID: \"a49e0328-71c1-43ce-80e5-83d0c0862cf9\") " pod="openshift-multus/multus-wpxd6" Jul 10 02:07:50.851912 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.807348 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a49e0328-71c1-43ce-80e5-83d0c0862cf9-cnibin\") pod \"multus-wpxd6\" (UID: \"a49e0328-71c1-43ce-80e5-83d0c0862cf9\") " pod="openshift-multus/multus-wpxd6" Jul 10 02:07:50.851912 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.807379 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a49e0328-71c1-43ce-80e5-83d0c0862cf9-multus-socket-dir-parent\") pod \"multus-wpxd6\" (UID: \"a49e0328-71c1-43ce-80e5-83d0c0862cf9\") " pod="openshift-multus/multus-wpxd6" Jul 10 02:07:50.851912 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.807407 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a49e0328-71c1-43ce-80e5-83d0c0862cf9-host-run-k8s-cni-cncf-io\") pod \"multus-wpxd6\" (UID: \"a49e0328-71c1-43ce-80e5-83d0c0862cf9\") " pod="openshift-multus/multus-wpxd6" Jul 10 02:07:50.851912 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.807450 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a49e0328-71c1-43ce-80e5-83d0c0862cf9-host-var-lib-cni-multus\") pod \"multus-wpxd6\" (UID: \"a49e0328-71c1-43ce-80e5-83d0c0862cf9\") " pod="openshift-multus/multus-wpxd6" Jul 10 02:07:50.853429 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.807485 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a49e0328-71c1-43ce-80e5-83d0c0862cf9-cni-binary-copy\") pod \"multus-wpxd6\" (UID: \"a49e0328-71c1-43ce-80e5-83d0c0862cf9\") " pod="openshift-multus/multus-wpxd6" Jul 10 02:07:50.853429 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.807510 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a49e0328-71c1-43ce-80e5-83d0c0862cf9-host-var-lib-kubelet\") pod \"multus-wpxd6\" (UID: \"a49e0328-71c1-43ce-80e5-83d0c0862cf9\") " pod="openshift-multus/multus-wpxd6" Jul 10 02:07:50.853429 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.807542 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pkdk\" (UniqueName: \"kubernetes.io/projected/a49e0328-71c1-43ce-80e5-83d0c0862cf9-kube-api-access-5pkdk\") pod \"multus-wpxd6\" (UID: \"a49e0328-71c1-43ce-80e5-83d0c0862cf9\") " pod="openshift-multus/multus-wpxd6" Jul 10 02:07:50.853429 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.807574 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a49e0328-71c1-43ce-80e5-83d0c0862cf9-multus-cni-dir\") pod \"multus-wpxd6\" (UID: \"a49e0328-71c1-43ce-80e5-83d0c0862cf9\") " pod="openshift-multus/multus-wpxd6" Jul 10 02:07:50.853429 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.807605 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a49e0328-71c1-43ce-80e5-83d0c0862cf9-os-release\") pod \"multus-wpxd6\" (UID: \"a49e0328-71c1-43ce-80e5-83d0c0862cf9\") " pod="openshift-multus/multus-wpxd6" Jul 10 02:07:50.853429 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.807642 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a49e0328-71c1-43ce-80e5-83d0c0862cf9-host-run-netns\") pod \"multus-wpxd6\" (UID: \"a49e0328-71c1-43ce-80e5-83d0c0862cf9\") " pod="openshift-multus/multus-wpxd6" Jul 10 02:07:50.853429 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.807665 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a49e0328-71c1-43ce-80e5-83d0c0862cf9-host-var-lib-cni-bin\") pod \"multus-wpxd6\" (UID: \"a49e0328-71c1-43ce-80e5-83d0c0862cf9\") " pod="openshift-multus/multus-wpxd6" Jul 10 02:07:50.853429 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.807686 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a49e0328-71c1-43ce-80e5-83d0c0862cf9-multus-daemon-config\") pod \"multus-wpxd6\" (UID: \"a49e0328-71c1-43ce-80e5-83d0c0862cf9\") " pod="openshift-multus/multus-wpxd6" Jul 10 02:07:50.853429 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.807708 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a49e0328-71c1-43ce-80e5-83d0c0862cf9-host-run-multus-certs\") pod \"multus-wpxd6\" (UID: \"a49e0328-71c1-43ce-80e5-83d0c0862cf9\") " pod="openshift-multus/multus-wpxd6" Jul 10 02:07:50.853429 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.807729 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a49e0328-71c1-43ce-80e5-83d0c0862cf9-etc-kubernetes\") pod \"multus-wpxd6\" (UID: \"a49e0328-71c1-43ce-80e5-83d0c0862cf9\") " pod="openshift-multus/multus-wpxd6" Jul 10 02:07:50.853429 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.807801 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a49e0328-71c1-43ce-80e5-83d0c0862cf9-hostroot\") pod \"multus-wpxd6\" (UID: \"a49e0328-71c1-43ce-80e5-83d0c0862cf9\") " pod="openshift-multus/multus-wpxd6" Jul 10 02:07:50.853429 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.807836 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a49e0328-71c1-43ce-80e5-83d0c0862cf9-multus-conf-dir\") pod \"multus-wpxd6\" (UID: \"a49e0328-71c1-43ce-80e5-83d0c0862cf9\") " pod="openshift-multus/multus-wpxd6" Jul 10 02:07:50.853429 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.807971 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vtbg2" Jul 10 02:07:50.853429 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.809895 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 10 02:07:50.853429 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.810104 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-x42mc\"" Jul 10 02:07:50.853429 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.810275 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 10 02:07:50.853429 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.810431 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 10 02:07:50.853429 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.811225 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-68dsx" Jul 10 02:07:50.854602 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.813166 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-f9csf" Jul 10 02:07:50.854602 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.815753 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8t46j" Jul 10 02:07:50.854602 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.817094 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 10 02:07:50.854602 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.817164 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qf5w9\"" Jul 10 02:07:50.854602 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.817411 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 10 02:07:50.854602 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.817496 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 10 02:07:50.854602 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.818484 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 10 02:07:50.854602 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.819043 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 10 02:07:50.854602 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.819269 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-z5f8w\"" Jul 10 02:07:50.854602 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.819310 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 10 02:07:50.854602 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.819679 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 10 02:07:50.854602 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.819733 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-58txz" Jul 10 02:07:50.854602 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.819745 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hblbw\"" Jul 10 02:07:50.854602 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.819939 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 10 02:07:50.854602 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.819970 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dbjd8\"" Jul 10 02:07:50.854602 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.820132 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 10 02:07:50.854602 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.820143 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 10 02:07:50.854602 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.820268 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 10 02:07:50.854602 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.820243 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2vqc6\"" Jul 10 02:07:50.854602 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.820440 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 10 02:07:50.854602 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.820492 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 10 02:07:50.854602 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.822212 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 10 02:07:50.854602 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.822590 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hfgs" Jul 10 02:07:50.854602 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:50.822764 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6hfgs" podUID="8bf086af-02b9-43f2-935f-474133fef4cf" Jul 10 02:07:50.854602 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.823067 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 10 02:07:50.854602 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.823498 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6lp6p\"" Jul 10 02:07:50.854602 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.823554 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 10 02:07:50.854602 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.824985 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ll898" Jul 10 02:07:50.856448 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:50.825049 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ll898" podUID="2fea8615-df10-4ccb-84cc-8d382f7bdafe" Jul 10 02:07:50.856448 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.828168 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcfzz" Jul 10 02:07:50.856448 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:50.828225 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcfzz" podUID="709370a6-295c-4330-bfa5-50ca602a28fb" Jul 10 02:07:50.856448 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.829444 2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 02:07:50.856448 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.833608 2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-nj5gx" Jul 10 02:07:50.856448 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.846643 2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-nj5gx" Jul 10 02:07:50.944487 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.893757 2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 10 02:07:50.944487 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.909019 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-run-ovn\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:50.944487 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.909060 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/29144f4e-ebaf-40b4-b9f1-9ee1fa371890-etc-modprobe-d\") pod \"tuned-68dsx\" (UID: \"29144f4e-ebaf-40b4-b9f1-9ee1fa371890\") " pod="openshift-cluster-node-tuning-operator/tuned-68dsx" Jul 10 02:07:50.944487 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.909090 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fad9e446-daca-42e4-a618-5defb2494fc3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-f9csf\" (UID: \"fad9e446-daca-42e4-a618-5defb2494fc3\") " pod="openshift-multus/multus-additional-cni-plugins-f9csf" Jul 10 02:07:50.944487 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.909129 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a49e0328-71c1-43ce-80e5-83d0c0862cf9-multus-daemon-config\") pod \"multus-wpxd6\" (UID: \"a49e0328-71c1-43ce-80e5-83d0c0862cf9\") " pod="openshift-multus/multus-wpxd6" Jul 10 02:07:50.944487 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.909156 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-ovnkube-script-lib\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:50.944487 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.909181 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/29144f4e-ebaf-40b4-b9f1-9ee1fa371890-etc-systemd\") pod \"tuned-68dsx\" (UID: \"29144f4e-ebaf-40b4-b9f1-9ee1fa371890\") " pod="openshift-cluster-node-tuning-operator/tuned-68dsx" Jul 10 02:07:50.944487 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.909216 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fad9e446-daca-42e4-a618-5defb2494fc3-os-release\") pod \"multus-additional-cni-plugins-f9csf\" (UID: \"fad9e446-daca-42e4-a618-5defb2494fc3\") " pod="openshift-multus/multus-additional-cni-plugins-f9csf" Jul 10 02:07:50.944487 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.909253 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f9v2\" (UniqueName: \"kubernetes.io/projected/fad9e446-daca-42e4-a618-5defb2494fc3-kube-api-access-7f9v2\") pod \"multus-additional-cni-plugins-f9csf\" (UID: \"fad9e446-daca-42e4-a618-5defb2494fc3\") " pod="openshift-multus/multus-additional-cni-plugins-f9csf" Jul 10 02:07:50.944487 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.909280 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8ccffaf3-b29f-4f58-ba37-5ab6f4992d19-hosts-file\") pod \"node-resolver-xgndm\" (UID: \"8ccffaf3-b29f-4f58-ba37-5ab6f4992d19\") " pod="openshift-dns/node-resolver-xgndm" Jul 10 02:07:50.944487 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.909305 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0fe32bca-a055-4658-b899-d22fbd705e41-socket-dir\") pod \"aws-ebs-csi-driver-node-vtbg2\" (UID: \"0fe32bca-a055-4658-b899-d22fbd705e41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vtbg2" Jul 10 02:07:50.944487 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.909331 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/54d67123-d94f-4ae2-8f94-ec576ba2e8b3-host-slash\") pod \"iptables-alerter-58txz\" (UID: \"54d67123-d94f-4ae2-8f94-ec576ba2e8b3\") " pod="openshift-network-operator/iptables-alerter-58txz" Jul 10 02:07:50.944487 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.909357 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-systemd-units\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:50.944487 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.909380 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/86176d1c-3214-46c1-91fe-94f087e0bda4-host\") pod \"node-ca-8t46j\" (UID: \"86176d1c-3214-46c1-91fe-94f087e0bda4\") " pod="openshift-image-registry/node-ca-8t46j" Jul 10 02:07:50.945195 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.909409 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a49e0328-71c1-43ce-80e5-83d0c0862cf9-cnibin\") pod \"multus-wpxd6\" (UID: \"a49e0328-71c1-43ce-80e5-83d0c0862cf9\") " pod="openshift-multus/multus-wpxd6" Jul 10 02:07:50.945195 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.909435 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/29144f4e-ebaf-40b4-b9f1-9ee1fa371890-etc-sysctl-conf\") pod \"tuned-68dsx\" (UID: \"29144f4e-ebaf-40b4-b9f1-9ee1fa371890\") " pod="openshift-cluster-node-tuning-operator/tuned-68dsx" Jul 10 02:07:50.945195 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.909460 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ccffaf3-b29f-4f58-ba37-5ab6f4992d19-tmp-dir\") pod \"node-resolver-xgndm\" (UID: \"8ccffaf3-b29f-4f58-ba37-5ab6f4992d19\") " pod="openshift-dns/node-resolver-xgndm" Jul 10 02:07:50.945195 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.909497 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-var-lib-openvswitch\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:50.945195 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.909526 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rl78\" (UniqueName: \"kubernetes.io/projected/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-kube-api-access-5rl78\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:50.945195 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.909555 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a49e0328-71c1-43ce-80e5-83d0c0862cf9-host-var-lib-cni-bin\") pod \"multus-wpxd6\" (UID: \"a49e0328-71c1-43ce-80e5-83d0c0862cf9\") " pod="openshift-multus/multus-wpxd6" Jul 10 02:07:50.945195 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.909580 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/29144f4e-ebaf-40b4-b9f1-9ee1fa371890-host\") pod \"tuned-68dsx\" (UID: \"29144f4e-ebaf-40b4-b9f1-9ee1fa371890\") " pod="openshift-cluster-node-tuning-operator/tuned-68dsx" Jul 10 02:07:50.945195 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.909604 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/29144f4e-ebaf-40b4-b9f1-9ee1fa371890-etc-tuned\") pod \"tuned-68dsx\" (UID: \"29144f4e-ebaf-40b4-b9f1-9ee1fa371890\") " pod="openshift-cluster-node-tuning-operator/tuned-68dsx" Jul 10 02:07:50.945195 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.909627 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29144f4e-ebaf-40b4-b9f1-9ee1fa371890-tmp\") pod \"tuned-68dsx\" (UID: \"29144f4e-ebaf-40b4-b9f1-9ee1fa371890\") " pod="openshift-cluster-node-tuning-operator/tuned-68dsx" Jul 10 02:07:50.945195 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.909651 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fad9e446-daca-42e4-a618-5defb2494fc3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f9csf\" (UID: \"fad9e446-daca-42e4-a618-5defb2494fc3\") " pod="openshift-multus/multus-additional-cni-plugins-f9csf" Jul 10 02:07:50.945195 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.909729 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a49e0328-71c1-43ce-80e5-83d0c0862cf9-multus-conf-dir\") pod \"multus-wpxd6\" (UID: \"a49e0328-71c1-43ce-80e5-83d0c0862cf9\") " pod="openshift-multus/multus-wpxd6" Jul 10 02:07:50.945195 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.909756 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-host-run-netns\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:50.945195 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.909782 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/29144f4e-ebaf-40b4-b9f1-9ee1fa371890-etc-sysconfig\") pod \"tuned-68dsx\" (UID: \"29144f4e-ebaf-40b4-b9f1-9ee1fa371890\") " pod="openshift-cluster-node-tuning-operator/tuned-68dsx" Jul 10 02:07:50.945195 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.909809 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0fe32bca-a055-4658-b899-d22fbd705e41-device-dir\") pod \"aws-ebs-csi-driver-node-vtbg2\" (UID: \"0fe32bca-a055-4658-b899-d22fbd705e41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vtbg2" Jul 10 02:07:50.946358 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.909836 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a49e0328-71c1-43ce-80e5-83d0c0862cf9-multus-socket-dir-parent\") pod \"multus-wpxd6\" (UID: \"a49e0328-71c1-43ce-80e5-83d0c0862cf9\") " pod="openshift-multus/multus-wpxd6" Jul 10 02:07:50.946358 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.909861 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-run-systemd\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:50.946358 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.909893 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/29144f4e-ebaf-40b4-b9f1-9ee1fa371890-etc-sysctl-d\") pod \"tuned-68dsx\" (UID: \"29144f4e-ebaf-40b4-b9f1-9ee1fa371890\") " pod="openshift-cluster-node-tuning-operator/tuned-68dsx" Jul 10 02:07:50.946358 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.909919 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fad9e446-daca-42e4-a618-5defb2494fc3-system-cni-dir\") pod \"multus-additional-cni-plugins-f9csf\" (UID: \"fad9e446-daca-42e4-a618-5defb2494fc3\") " pod="openshift-multus/multus-additional-cni-plugins-f9csf" Jul 10 02:07:50.946358 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.909944 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/86176d1c-3214-46c1-91fe-94f087e0bda4-serviceca\") pod \"node-ca-8t46j\" (UID: \"86176d1c-3214-46c1-91fe-94f087e0bda4\") " pod="openshift-image-registry/node-ca-8t46j" Jul 10 02:07:50.946358 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.909972 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfmw4\" (UniqueName: \"kubernetes.io/projected/2fea8615-df10-4ccb-84cc-8d382f7bdafe-kube-api-access-lfmw4\") pod \"network-check-target-ll898\" (UID: \"2fea8615-df10-4ccb-84cc-8d382f7bdafe\") " pod="openshift-network-diagnostics/network-check-target-ll898" Jul 10 02:07:50.946358 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.910016 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmb5f\" (UniqueName: \"kubernetes.io/projected/0fe32bca-a055-4658-b899-d22fbd705e41-kube-api-access-lmb5f\") pod \"aws-ebs-csi-driver-node-vtbg2\" (UID: \"0fe32bca-a055-4658-b899-d22fbd705e41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vtbg2" Jul 10 02:07:50.946358 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.910048 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a49e0328-71c1-43ce-80e5-83d0c0862cf9-host-var-lib-kubelet\") pod \"multus-wpxd6\" (UID: \"a49e0328-71c1-43ce-80e5-83d0c0862cf9\") " pod="openshift-multus/multus-wpxd6" Jul 10 02:07:50.946358 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.910076 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/54d67123-d94f-4ae2-8f94-ec576ba2e8b3-iptables-alerter-script\") pod \"iptables-alerter-58txz\" (UID: \"54d67123-d94f-4ae2-8f94-ec576ba2e8b3\") " pod="openshift-network-operator/iptables-alerter-58txz" Jul 10 02:07:50.946358 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.910102 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpdwc\" (UniqueName: \"kubernetes.io/projected/86176d1c-3214-46c1-91fe-94f087e0bda4-kube-api-access-wpdwc\") pod \"node-ca-8t46j\" (UID: \"86176d1c-3214-46c1-91fe-94f087e0bda4\") " pod="openshift-image-registry/node-ca-8t46j" Jul 10 02:07:50.946358 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.910128 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl5f2\" (UniqueName: \"kubernetes.io/projected/8ccffaf3-b29f-4f58-ba37-5ab6f4992d19-kube-api-access-nl5f2\") pod \"node-resolver-xgndm\" (UID: \"8ccffaf3-b29f-4f58-ba37-5ab6f4992d19\") " pod="openshift-dns/node-resolver-xgndm" Jul 10 02:07:50.946358 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.910156 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a49e0328-71c1-43ce-80e5-83d0c0862cf9-multus-cni-dir\") pod \"multus-wpxd6\" (UID: \"a49e0328-71c1-43ce-80e5-83d0c0862cf9\") " pod="openshift-multus/multus-wpxd6" Jul 10 02:07:50.946358 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.910182 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a49e0328-71c1-43ce-80e5-83d0c0862cf9-os-release\") pod \"multus-wpxd6\" (UID: \"a49e0328-71c1-43ce-80e5-83d0c0862cf9\") " pod="openshift-multus/multus-wpxd6" Jul 10 02:07:50.946358 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.910223 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a49e0328-71c1-43ce-80e5-83d0c0862cf9-host-run-netns\") pod \"multus-wpxd6\" (UID: \"a49e0328-71c1-43ce-80e5-83d0c0862cf9\") " pod="openshift-multus/multus-wpxd6" Jul 10 02:07:50.947279 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.910251 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9g42\" (UniqueName: \"kubernetes.io/projected/54d67123-d94f-4ae2-8f94-ec576ba2e8b3-kube-api-access-q9g42\") pod \"iptables-alerter-58txz\" (UID: \"54d67123-d94f-4ae2-8f94-ec576ba2e8b3\") " pod="openshift-network-operator/iptables-alerter-58txz" Jul 10 02:07:50.947279 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.910283 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-etc-openvswitch\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:50.947279 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.910312 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:50.947279 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.910340 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-env-overrides\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:50.947279 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.910385 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-ovn-node-metrics-cert\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:50.947279 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.910413 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bf086af-02b9-43f2-935f-474133fef4cf-metrics-certs\") pod \"network-metrics-daemon-6hfgs\" (UID: \"8bf086af-02b9-43f2-935f-474133fef4cf\") " pod="openshift-multus/network-metrics-daemon-6hfgs" Jul 10 02:07:50.947279 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.910440 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/709370a6-295c-4330-bfa5-50ca602a28fb-dbus\") pod \"global-pull-secret-syncer-pcfzz\" (UID: \"709370a6-295c-4330-bfa5-50ca602a28fb\") " pod="kube-system/global-pull-secret-syncer-pcfzz" Jul 10 02:07:50.947279 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.910466 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/452d0117-6775-4e71-acc6-4e2fc639e369-konnectivity-ca\") pod \"konnectivity-agent-cprsf\" (UID: \"452d0117-6775-4e71-acc6-4e2fc639e369\") " pod="kube-system/konnectivity-agent-cprsf" Jul 10 02:07:50.947279 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.910497 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-log-socket\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:50.947279 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.910524 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/29144f4e-ebaf-40b4-b9f1-9ee1fa371890-run\") pod \"tuned-68dsx\" (UID: \"29144f4e-ebaf-40b4-b9f1-9ee1fa371890\") " pod="openshift-cluster-node-tuning-operator/tuned-68dsx" Jul 10 02:07:50.947279 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.910552 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/29144f4e-ebaf-40b4-b9f1-9ee1fa371890-lib-modules\") pod \"tuned-68dsx\" (UID: \"29144f4e-ebaf-40b4-b9f1-9ee1fa371890\") " pod="openshift-cluster-node-tuning-operator/tuned-68dsx" Jul 10 02:07:50.947279 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.910580 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fad9e446-daca-42e4-a618-5defb2494fc3-cnibin\") pod \"multus-additional-cni-plugins-f9csf\" (UID: \"fad9e446-daca-42e4-a618-5defb2494fc3\") " pod="openshift-multus/multus-additional-cni-plugins-f9csf" Jul 10 02:07:50.947279 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.910649 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fad9e446-daca-42e4-a618-5defb2494fc3-cni-binary-copy\") pod \"multus-additional-cni-plugins-f9csf\" (UID: \"fad9e446-daca-42e4-a618-5defb2494fc3\") " pod="openshift-multus/multus-additional-cni-plugins-f9csf" Jul 10 02:07:50.948653 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.910699 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a49e0328-71c1-43ce-80e5-83d0c0862cf9-host-run-k8s-cni-cncf-io\") pod \"multus-wpxd6\" (UID: \"a49e0328-71c1-43ce-80e5-83d0c0862cf9\") " pod="openshift-multus/multus-wpxd6" Jul 10 02:07:50.948653 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.910728 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/709370a6-295c-4330-bfa5-50ca602a28fb-kubelet-config\") pod \"global-pull-secret-syncer-pcfzz\" (UID: \"709370a6-295c-4330-bfa5-50ca602a28fb\") " pod="kube-system/global-pull-secret-syncer-pcfzz" Jul 10 02:07:50.948653 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.910754 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-run-openvswitch\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:50.948653 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.910780 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-ovnkube-config\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:50.948653 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.910812 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29144f4e-ebaf-40b4-b9f1-9ee1fa371890-etc-kubernetes\") pod \"tuned-68dsx\" (UID: \"29144f4e-ebaf-40b4-b9f1-9ee1fa371890\") " pod="openshift-cluster-node-tuning-operator/tuned-68dsx" Jul 10 02:07:50.948653 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.910836 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0fe32bca-a055-4658-b899-d22fbd705e41-sys-fs\") pod \"aws-ebs-csi-driver-node-vtbg2\" (UID: \"0fe32bca-a055-4658-b899-d22fbd705e41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vtbg2" Jul 10 02:07:50.948653 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.910858 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-host-slash\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:50.948653 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.910881 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0fe32bca-a055-4658-b899-d22fbd705e41-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vtbg2\" (UID: \"0fe32bca-a055-4658-b899-d22fbd705e41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vtbg2" Jul 10 02:07:50.948653 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.910912 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a49e0328-71c1-43ce-80e5-83d0c0862cf9-host-run-multus-certs\") pod \"multus-wpxd6\" (UID: \"a49e0328-71c1-43ce-80e5-83d0c0862cf9\") " pod="openshift-multus/multus-wpxd6" Jul 10 02:07:50.948653 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.910924 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a49e0328-71c1-43ce-80e5-83d0c0862cf9-multus-socket-dir-parent\") pod \"multus-wpxd6\" (UID: \"a49e0328-71c1-43ce-80e5-83d0c0862cf9\") " pod="openshift-multus/multus-wpxd6" Jul 10 02:07:50.948653 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.910939 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a49e0328-71c1-43ce-80e5-83d0c0862cf9-etc-kubernetes\") pod \"multus-wpxd6\" (UID: \"a49e0328-71c1-43ce-80e5-83d0c0862cf9\") " pod="openshift-multus/multus-wpxd6" Jul 10 02:07:50.948653 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.910967 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpskb\" (UniqueName: \"kubernetes.io/projected/8bf086af-02b9-43f2-935f-474133fef4cf-kube-api-access-dpskb\") pod \"network-metrics-daemon-6hfgs\" (UID: \"8bf086af-02b9-43f2-935f-474133fef4cf\") " pod="openshift-multus/network-metrics-daemon-6hfgs" Jul 10 02:07:50.948653 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.911008 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-host-kubelet\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:50.948653 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.911038 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0fe32bca-a055-4658-b899-d22fbd705e41-registration-dir\") pod \"aws-ebs-csi-driver-node-vtbg2\" (UID: \"0fe32bca-a055-4658-b899-d22fbd705e41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vtbg2" Jul 10 02:07:50.949605 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.911348 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a49e0328-71c1-43ce-80e5-83d0c0862cf9-hostroot\") pod \"multus-wpxd6\" (UID: \"a49e0328-71c1-43ce-80e5-83d0c0862cf9\") " pod="openshift-multus/multus-wpxd6" Jul 10 02:07:50.949605 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.911409 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/709370a6-295c-4330-bfa5-50ca602a28fb-original-pull-secret\") pod \"global-pull-secret-syncer-pcfzz\" (UID: \"709370a6-295c-4330-bfa5-50ca602a28fb\") " pod="kube-system/global-pull-secret-syncer-pcfzz" Jul 10 02:07:50.949605 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.911450 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-host-cni-netd\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:50.949605 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.911484 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/29144f4e-ebaf-40b4-b9f1-9ee1fa371890-sys\") pod \"tuned-68dsx\" (UID: \"29144f4e-ebaf-40b4-b9f1-9ee1fa371890\") " pod="openshift-cluster-node-tuning-operator/tuned-68dsx" Jul 10 02:07:50.949605 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.911542 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29144f4e-ebaf-40b4-b9f1-9ee1fa371890-var-lib-kubelet\") pod \"tuned-68dsx\" (UID: \"29144f4e-ebaf-40b4-b9f1-9ee1fa371890\") " pod="openshift-cluster-node-tuning-operator/tuned-68dsx" Jul 10 02:07:50.949605 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.912508 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a49e0328-71c1-43ce-80e5-83d0c0862cf9-multus-daemon-config\") pod \"multus-wpxd6\" (UID: \"a49e0328-71c1-43ce-80e5-83d0c0862cf9\") " pod="openshift-multus/multus-wpxd6" Jul 10 02:07:50.949605 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.913060 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a49e0328-71c1-43ce-80e5-83d0c0862cf9-host-run-multus-certs\") pod \"multus-wpxd6\" (UID: \"a49e0328-71c1-43ce-80e5-83d0c0862cf9\") " pod="openshift-multus/multus-wpxd6" Jul 10 02:07:50.949605 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.913064 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a49e0328-71c1-43ce-80e5-83d0c0862cf9-host-var-lib-cni-bin\") pod \"multus-wpxd6\" (UID: \"a49e0328-71c1-43ce-80e5-83d0c0862cf9\") " pod="openshift-multus/multus-wpxd6" Jul 10 02:07:50.949605 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.913130 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a49e0328-71c1-43ce-80e5-83d0c0862cf9-host-var-lib-kubelet\") pod \"multus-wpxd6\" (UID: \"a49e0328-71c1-43ce-80e5-83d0c0862cf9\") " pod="openshift-multus/multus-wpxd6" Jul 10 02:07:50.949605 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.913176 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a49e0328-71c1-43ce-80e5-83d0c0862cf9-etc-kubernetes\") pod \"multus-wpxd6\" (UID: \"a49e0328-71c1-43ce-80e5-83d0c0862cf9\") " pod="openshift-multus/multus-wpxd6" Jul 10 02:07:50.949605 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.913196 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a49e0328-71c1-43ce-80e5-83d0c0862cf9-host-run-k8s-cni-cncf-io\") pod \"multus-wpxd6\" (UID: \"a49e0328-71c1-43ce-80e5-83d0c0862cf9\") " pod="openshift-multus/multus-wpxd6" Jul 10 02:07:50.949605 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.913314 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjv5d\" (UniqueName: \"kubernetes.io/projected/29144f4e-ebaf-40b4-b9f1-9ee1fa371890-kube-api-access-tjv5d\") pod \"tuned-68dsx\" (UID: \"29144f4e-ebaf-40b4-b9f1-9ee1fa371890\") " pod="openshift-cluster-node-tuning-operator/tuned-68dsx" Jul 10 02:07:50.949605 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.913332 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a49e0328-71c1-43ce-80e5-83d0c0862cf9-cnibin\") pod \"multus-wpxd6\" (UID: \"a49e0328-71c1-43ce-80e5-83d0c0862cf9\") " pod="openshift-multus/multus-wpxd6" Jul 10 02:07:50.949605 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.913342 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a49e0328-71c1-43ce-80e5-83d0c0862cf9-multus-cni-dir\") pod \"multus-wpxd6\" (UID: \"a49e0328-71c1-43ce-80e5-83d0c0862cf9\") " pod="openshift-multus/multus-wpxd6" Jul 10 02:07:50.949605 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.913364 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a49e0328-71c1-43ce-80e5-83d0c0862cf9-system-cni-dir\") pod \"multus-wpxd6\" (UID: \"a49e0328-71c1-43ce-80e5-83d0c0862cf9\") " pod="openshift-multus/multus-wpxd6" Jul 10 02:07:50.951237 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.913396 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a49e0328-71c1-43ce-80e5-83d0c0862cf9-host-run-netns\") pod \"multus-wpxd6\" (UID: \"a49e0328-71c1-43ce-80e5-83d0c0862cf9\") " pod="openshift-multus/multus-wpxd6" Jul 10 02:07:50.951237 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.913409 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a49e0328-71c1-43ce-80e5-83d0c0862cf9-host-var-lib-cni-multus\") pod \"multus-wpxd6\" (UID: \"a49e0328-71c1-43ce-80e5-83d0c0862cf9\") " pod="openshift-multus/multus-wpxd6" Jul 10 02:07:50.951237 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.913469 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a49e0328-71c1-43ce-80e5-83d0c0862cf9-hostroot\") pod \"multus-wpxd6\" (UID: \"a49e0328-71c1-43ce-80e5-83d0c0862cf9\") " pod="openshift-multus/multus-wpxd6" Jul 10 02:07:50.951237 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.913541 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a49e0328-71c1-43ce-80e5-83d0c0862cf9-os-release\") pod \"multus-wpxd6\" (UID: \"a49e0328-71c1-43ce-80e5-83d0c0862cf9\") " pod="openshift-multus/multus-wpxd6" Jul 10 02:07:50.951237 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.913601 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a49e0328-71c1-43ce-80e5-83d0c0862cf9-system-cni-dir\") pod \"multus-wpxd6\" (UID: \"a49e0328-71c1-43ce-80e5-83d0c0862cf9\") " pod="openshift-multus/multus-wpxd6" Jul 10 02:07:50.951237 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.913811 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-node-log\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:50.951237 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.913839 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a49e0328-71c1-43ce-80e5-83d0c0862cf9-host-var-lib-cni-multus\") pod \"multus-wpxd6\" (UID: \"a49e0328-71c1-43ce-80e5-83d0c0862cf9\") " pod="openshift-multus/multus-wpxd6" Jul 10 02:07:50.951237 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.913864 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-host-run-ovn-kubernetes\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:50.951237 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.913888 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a49e0328-71c1-43ce-80e5-83d0c0862cf9-multus-conf-dir\") pod \"multus-wpxd6\" (UID: \"a49e0328-71c1-43ce-80e5-83d0c0862cf9\") " pod="openshift-multus/multus-wpxd6" Jul 10 02:07:50.951237 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.913911 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-host-cni-bin\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:50.951237 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.913955 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fad9e446-daca-42e4-a618-5defb2494fc3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-f9csf\" (UID: \"fad9e446-daca-42e4-a618-5defb2494fc3\") " pod="openshift-multus/multus-additional-cni-plugins-f9csf" Jul 10 02:07:50.951237 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.914002 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0fe32bca-a055-4658-b899-d22fbd705e41-etc-selinux\") pod \"aws-ebs-csi-driver-node-vtbg2\" (UID: \"0fe32bca-a055-4658-b899-d22fbd705e41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vtbg2" Jul 10 02:07:50.951237 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.914048 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a49e0328-71c1-43ce-80e5-83d0c0862cf9-cni-binary-copy\") pod \"multus-wpxd6\" (UID: \"a49e0328-71c1-43ce-80e5-83d0c0862cf9\") " pod="openshift-multus/multus-wpxd6" Jul 10 02:07:50.951237 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.914091 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pkdk\" (UniqueName: \"kubernetes.io/projected/a49e0328-71c1-43ce-80e5-83d0c0862cf9-kube-api-access-5pkdk\") pod \"multus-wpxd6\" (UID: \"a49e0328-71c1-43ce-80e5-83d0c0862cf9\") " pod="openshift-multus/multus-wpxd6" Jul 10 02:07:50.951237 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.914129 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/452d0117-6775-4e71-acc6-4e2fc639e369-agent-certs\") pod \"konnectivity-agent-cprsf\" (UID: \"452d0117-6775-4e71-acc6-4e2fc639e369\") " pod="kube-system/konnectivity-agent-cprsf" Jul 10 02:07:50.952190 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.914615 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a49e0328-71c1-43ce-80e5-83d0c0862cf9-cni-binary-copy\") pod \"multus-wpxd6\" (UID: \"a49e0328-71c1-43ce-80e5-83d0c0862cf9\") " pod="openshift-multus/multus-wpxd6" Jul 10 02:07:50.952190 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.924023 2566 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 10 02:07:50.952190 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:50.928857 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pkdk\" (UniqueName: \"kubernetes.io/projected/a49e0328-71c1-43ce-80e5-83d0c0862cf9-kube-api-access-5pkdk\") pod \"multus-wpxd6\" (UID: \"a49e0328-71c1-43ce-80e5-83d0c0862cf9\") " pod="openshift-multus/multus-wpxd6" Jul 10 02:07:51.058994 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.012278 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-65.ec2.internal" event={"ID":"305afbb58c749f5158dc47cb0c52154f","Type":"ContainerStarted","Data":"09853460d34b6a91f2b87bf216f6aa73e5f81ac3a370e05da0d4466e57511872"} Jul 10 02:07:51.058994 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.013868 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-65.ec2.internal" event={"ID":"aff7f4ed066dfa82229540d0126815cc","Type":"ContainerStarted","Data":"29ebf9b96422ba536ee97f44a7c633d86635e0588351d4347224cf74eadbae8d"} Jul 10 02:07:51.058994 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.030367 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-run-systemd\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:51.058994 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.030400 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/29144f4e-ebaf-40b4-b9f1-9ee1fa371890-etc-sysctl-d\") pod \"tuned-68dsx\" (UID: \"29144f4e-ebaf-40b4-b9f1-9ee1fa371890\") " pod="openshift-cluster-node-tuning-operator/tuned-68dsx" Jul 10 02:07:51.058994 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.030420 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fad9e446-daca-42e4-a618-5defb2494fc3-system-cni-dir\") pod \"multus-additional-cni-plugins-f9csf\" (UID: \"fad9e446-daca-42e4-a618-5defb2494fc3\") " pod="openshift-multus/multus-additional-cni-plugins-f9csf" Jul 10 02:07:51.058994 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.030437 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/86176d1c-3214-46c1-91fe-94f087e0bda4-serviceca\") pod \"node-ca-8t46j\" (UID: \"86176d1c-3214-46c1-91fe-94f087e0bda4\") " pod="openshift-image-registry/node-ca-8t46j" Jul 10 02:07:51.058994 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.030456 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfmw4\" (UniqueName: \"kubernetes.io/projected/2fea8615-df10-4ccb-84cc-8d382f7bdafe-kube-api-access-lfmw4\") pod \"network-check-target-ll898\" (UID: \"2fea8615-df10-4ccb-84cc-8d382f7bdafe\") " pod="openshift-network-diagnostics/network-check-target-ll898" Jul 10 02:07:51.058994 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.030504 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-run-systemd\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:51.058994 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.030515 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fad9e446-daca-42e4-a618-5defb2494fc3-system-cni-dir\") pod \"multus-additional-cni-plugins-f9csf\" (UID: \"fad9e446-daca-42e4-a618-5defb2494fc3\") " pod="openshift-multus/multus-additional-cni-plugins-f9csf" Jul 10 02:07:51.058994 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.030552 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmb5f\" (UniqueName: \"kubernetes.io/projected/0fe32bca-a055-4658-b899-d22fbd705e41-kube-api-access-lmb5f\") pod \"aws-ebs-csi-driver-node-vtbg2\" (UID: \"0fe32bca-a055-4658-b899-d22fbd705e41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vtbg2" Jul 10 02:07:51.058994 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.030586 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/54d67123-d94f-4ae2-8f94-ec576ba2e8b3-iptables-alerter-script\") pod \"iptables-alerter-58txz\" (UID: \"54d67123-d94f-4ae2-8f94-ec576ba2e8b3\") " pod="openshift-network-operator/iptables-alerter-58txz" Jul 10 02:07:51.058994 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.030601 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/29144f4e-ebaf-40b4-b9f1-9ee1fa371890-etc-sysctl-d\") pod \"tuned-68dsx\" (UID: \"29144f4e-ebaf-40b4-b9f1-9ee1fa371890\") " pod="openshift-cluster-node-tuning-operator/tuned-68dsx" Jul 10 02:07:51.058994 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.030635 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpdwc\" (UniqueName: \"kubernetes.io/projected/86176d1c-3214-46c1-91fe-94f087e0bda4-kube-api-access-wpdwc\") pod \"node-ca-8t46j\" (UID: \"86176d1c-3214-46c1-91fe-94f087e0bda4\") " pod="openshift-image-registry/node-ca-8t46j" Jul 10 02:07:51.058994 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.030665 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl5f2\" (UniqueName: \"kubernetes.io/projected/8ccffaf3-b29f-4f58-ba37-5ab6f4992d19-kube-api-access-nl5f2\") pod \"node-resolver-xgndm\" (UID: \"8ccffaf3-b29f-4f58-ba37-5ab6f4992d19\") " pod="openshift-dns/node-resolver-xgndm" Jul 10 02:07:51.061050 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.031043 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9g42\" (UniqueName: \"kubernetes.io/projected/54d67123-d94f-4ae2-8f94-ec576ba2e8b3-kube-api-access-q9g42\") pod \"iptables-alerter-58txz\" (UID: \"54d67123-d94f-4ae2-8f94-ec576ba2e8b3\") " pod="openshift-network-operator/iptables-alerter-58txz" Jul 10 02:07:51.061050 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.031089 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-etc-openvswitch\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:51.061050 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.031045 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/86176d1c-3214-46c1-91fe-94f087e0bda4-serviceca\") pod \"node-ca-8t46j\" (UID: \"86176d1c-3214-46c1-91fe-94f087e0bda4\") " pod="openshift-image-registry/node-ca-8t46j" Jul 10 02:07:51.061050 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.031128 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:51.061050 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.031156 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-env-overrides\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:51.061050 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.031177 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-etc-openvswitch\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:51.061050 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.031190 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-ovn-node-metrics-cert\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:51.061050 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.031216 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bf086af-02b9-43f2-935f-474133fef4cf-metrics-certs\") pod \"network-metrics-daemon-6hfgs\" (UID: \"8bf086af-02b9-43f2-935f-474133fef4cf\") " pod="openshift-multus/network-metrics-daemon-6hfgs" Jul 10 02:07:51.061050 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.031222 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:51.061050 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.031247 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/709370a6-295c-4330-bfa5-50ca602a28fb-dbus\") pod \"global-pull-secret-syncer-pcfzz\" (UID: \"709370a6-295c-4330-bfa5-50ca602a28fb\") " pod="kube-system/global-pull-secret-syncer-pcfzz" Jul 10 02:07:51.061050 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.031274 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/452d0117-6775-4e71-acc6-4e2fc639e369-konnectivity-ca\") pod \"konnectivity-agent-cprsf\" (UID: \"452d0117-6775-4e71-acc6-4e2fc639e369\") " pod="kube-system/konnectivity-agent-cprsf" Jul 10 02:07:51.061050 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.031306 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-log-socket\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:51.061050 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.031332 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/29144f4e-ebaf-40b4-b9f1-9ee1fa371890-run\") pod \"tuned-68dsx\" (UID: \"29144f4e-ebaf-40b4-b9f1-9ee1fa371890\") " pod="openshift-cluster-node-tuning-operator/tuned-68dsx" Jul 10 02:07:51.061050 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.031357 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/29144f4e-ebaf-40b4-b9f1-9ee1fa371890-lib-modules\") pod \"tuned-68dsx\" (UID: \"29144f4e-ebaf-40b4-b9f1-9ee1fa371890\") " pod="openshift-cluster-node-tuning-operator/tuned-68dsx" Jul 10 02:07:51.061987 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.031388 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fad9e446-daca-42e4-a618-5defb2494fc3-cnibin\") pod \"multus-additional-cni-plugins-f9csf\" (UID: \"fad9e446-daca-42e4-a618-5defb2494fc3\") " pod="openshift-multus/multus-additional-cni-plugins-f9csf" Jul 10 02:07:51.061987 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.031414 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fad9e446-daca-42e4-a618-5defb2494fc3-cni-binary-copy\") pod \"multus-additional-cni-plugins-f9csf\" (UID: \"fad9e446-daca-42e4-a618-5defb2494fc3\") " pod="openshift-multus/multus-additional-cni-plugins-f9csf" Jul 10 02:07:51.061987 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.031449 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/709370a6-295c-4330-bfa5-50ca602a28fb-kubelet-config\") pod \"global-pull-secret-syncer-pcfzz\" (UID: \"709370a6-295c-4330-bfa5-50ca602a28fb\") " pod="kube-system/global-pull-secret-syncer-pcfzz" Jul 10 02:07:51.061987 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.031630 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-env-overrides\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:51.061987 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.031696 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-log-socket\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:51.061987 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.031731 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/709370a6-295c-4330-bfa5-50ca602a28fb-dbus\") pod \"global-pull-secret-syncer-pcfzz\" (UID: \"709370a6-295c-4330-bfa5-50ca602a28fb\") " pod="kube-system/global-pull-secret-syncer-pcfzz" Jul 10 02:07:51.061987 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:51.031818 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 02:07:51.061987 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:51.031887 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bf086af-02b9-43f2-935f-474133fef4cf-metrics-certs podName:8bf086af-02b9-43f2-935f-474133fef4cf nodeName:}" failed. No retries permitted until 2026-07-10 02:07:51.531866244 +0000 UTC m=+3.108143328 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8bf086af-02b9-43f2-935f-474133fef4cf-metrics-certs") pod "network-metrics-daemon-6hfgs" (UID: "8bf086af-02b9-43f2-935f-474133fef4cf") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 02:07:51.061987 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.032093 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/29144f4e-ebaf-40b4-b9f1-9ee1fa371890-lib-modules\") pod \"tuned-68dsx\" (UID: \"29144f4e-ebaf-40b4-b9f1-9ee1fa371890\") " pod="openshift-cluster-node-tuning-operator/tuned-68dsx" Jul 10 02:07:51.061987 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.032164 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/29144f4e-ebaf-40b4-b9f1-9ee1fa371890-run\") pod \"tuned-68dsx\" (UID: \"29144f4e-ebaf-40b4-b9f1-9ee1fa371890\") " pod="openshift-cluster-node-tuning-operator/tuned-68dsx" Jul 10 02:07:51.061987 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.032212 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fad9e446-daca-42e4-a618-5defb2494fc3-cnibin\") pod \"multus-additional-cni-plugins-f9csf\" (UID: \"fad9e446-daca-42e4-a618-5defb2494fc3\") " pod="openshift-multus/multus-additional-cni-plugins-f9csf" Jul 10 02:07:51.061987 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.032804 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fad9e446-daca-42e4-a618-5defb2494fc3-cni-binary-copy\") pod \"multus-additional-cni-plugins-f9csf\" (UID: \"fad9e446-daca-42e4-a618-5defb2494fc3\") " pod="openshift-multus/multus-additional-cni-plugins-f9csf" Jul 10 02:07:51.061987 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.033183 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/452d0117-6775-4e71-acc6-4e2fc639e369-konnectivity-ca\") pod \"konnectivity-agent-cprsf\" (UID: \"452d0117-6775-4e71-acc6-4e2fc639e369\") " pod="kube-system/konnectivity-agent-cprsf" Jul 10 02:07:51.061987 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.033440 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/54d67123-d94f-4ae2-8f94-ec576ba2e8b3-iptables-alerter-script\") pod \"iptables-alerter-58txz\" (UID: \"54d67123-d94f-4ae2-8f94-ec576ba2e8b3\") " pod="openshift-network-operator/iptables-alerter-58txz" Jul 10 02:07:51.063327 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.033451 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/709370a6-295c-4330-bfa5-50ca602a28fb-kubelet-config\") pod \"global-pull-secret-syncer-pcfzz\" (UID: \"709370a6-295c-4330-bfa5-50ca602a28fb\") " pod="kube-system/global-pull-secret-syncer-pcfzz" Jul 10 02:07:51.063327 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.034653 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-run-openvswitch\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:51.063327 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.034705 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-ovnkube-config\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:51.063327 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.034733 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29144f4e-ebaf-40b4-b9f1-9ee1fa371890-etc-kubernetes\") pod \"tuned-68dsx\" (UID: \"29144f4e-ebaf-40b4-b9f1-9ee1fa371890\") " pod="openshift-cluster-node-tuning-operator/tuned-68dsx" Jul 10 02:07:51.063327 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.034757 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-run-openvswitch\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:51.063327 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.034764 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0fe32bca-a055-4658-b899-d22fbd705e41-sys-fs\") pod \"aws-ebs-csi-driver-node-vtbg2\" (UID: \"0fe32bca-a055-4658-b899-d22fbd705e41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vtbg2" Jul 10 02:07:51.063327 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.034817 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-host-slash\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:51.063327 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.034827 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0fe32bca-a055-4658-b899-d22fbd705e41-sys-fs\") pod \"aws-ebs-csi-driver-node-vtbg2\" (UID: \"0fe32bca-a055-4658-b899-d22fbd705e41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vtbg2" Jul 10 02:07:51.063327 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.034845 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0fe32bca-a055-4658-b899-d22fbd705e41-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vtbg2\" (UID: \"0fe32bca-a055-4658-b899-d22fbd705e41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vtbg2" Jul 10 02:07:51.063327 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.034905 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0fe32bca-a055-4658-b899-d22fbd705e41-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vtbg2\" (UID: \"0fe32bca-a055-4658-b899-d22fbd705e41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vtbg2" Jul 10 02:07:51.063327 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.034908 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29144f4e-ebaf-40b4-b9f1-9ee1fa371890-etc-kubernetes\") pod \"tuned-68dsx\" (UID: \"29144f4e-ebaf-40b4-b9f1-9ee1fa371890\") " pod="openshift-cluster-node-tuning-operator/tuned-68dsx" Jul 10 02:07:51.063327 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.034947 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-ovn-node-metrics-cert\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:51.063327 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.034968 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpskb\" (UniqueName: \"kubernetes.io/projected/8bf086af-02b9-43f2-935f-474133fef4cf-kube-api-access-dpskb\") pod \"network-metrics-daemon-6hfgs\" (UID: \"8bf086af-02b9-43f2-935f-474133fef4cf\") " pod="openshift-multus/network-metrics-daemon-6hfgs" Jul 10 02:07:51.063327 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.034951 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-host-slash\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:51.064006 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.034995 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-host-kubelet\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:51.064006 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.035021 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0fe32bca-a055-4658-b899-d22fbd705e41-registration-dir\") pod \"aws-ebs-csi-driver-node-vtbg2\" (UID: \"0fe32bca-a055-4658-b899-d22fbd705e41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vtbg2" Jul 10 02:07:51.064006 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.035048 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/709370a6-295c-4330-bfa5-50ca602a28fb-original-pull-secret\") pod \"global-pull-secret-syncer-pcfzz\" (UID: \"709370a6-295c-4330-bfa5-50ca602a28fb\") " pod="kube-system/global-pull-secret-syncer-pcfzz" Jul 10 02:07:51.064006 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.035078 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-host-cni-netd\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:51.064006 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.035064 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-host-kubelet\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:51.064006 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.035099 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0fe32bca-a055-4658-b899-d22fbd705e41-registration-dir\") pod \"aws-ebs-csi-driver-node-vtbg2\" (UID: \"0fe32bca-a055-4658-b899-d22fbd705e41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vtbg2" Jul 10 02:07:51.064006 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.035108 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/29144f4e-ebaf-40b4-b9f1-9ee1fa371890-sys\") pod \"tuned-68dsx\" (UID: \"29144f4e-ebaf-40b4-b9f1-9ee1fa371890\") " pod="openshift-cluster-node-tuning-operator/tuned-68dsx" Jul 10 02:07:51.064006 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.035151 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29144f4e-ebaf-40b4-b9f1-9ee1fa371890-var-lib-kubelet\") pod \"tuned-68dsx\" (UID: \"29144f4e-ebaf-40b4-b9f1-9ee1fa371890\") " pod="openshift-cluster-node-tuning-operator/tuned-68dsx" Jul 10 02:07:51.064006 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.035161 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/29144f4e-ebaf-40b4-b9f1-9ee1fa371890-sys\") pod \"tuned-68dsx\" (UID: \"29144f4e-ebaf-40b4-b9f1-9ee1fa371890\") " pod="openshift-cluster-node-tuning-operator/tuned-68dsx" Jul 10 02:07:51.064006 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.035179 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjv5d\" (UniqueName: \"kubernetes.io/projected/29144f4e-ebaf-40b4-b9f1-9ee1fa371890-kube-api-access-tjv5d\") pod \"tuned-68dsx\" (UID: \"29144f4e-ebaf-40b4-b9f1-9ee1fa371890\") " pod="openshift-cluster-node-tuning-operator/tuned-68dsx" Jul 10 02:07:51.064006 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:51.035188 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 02:07:51.064006 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.035210 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-node-log\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:51.064006 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.035201 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-host-cni-netd\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:51.064006 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.035241 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-host-run-ovn-kubernetes\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:51.065158 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:51.035256 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/709370a6-295c-4330-bfa5-50ca602a28fb-original-pull-secret podName:709370a6-295c-4330-bfa5-50ca602a28fb nodeName:}" failed. No retries permitted until 2026-07-10 02:07:51.535237633 +0000 UTC m=+3.111514709 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/709370a6-295c-4330-bfa5-50ca602a28fb-original-pull-secret") pod "global-pull-secret-syncer-pcfzz" (UID: "709370a6-295c-4330-bfa5-50ca602a28fb") : object "kube-system"/"original-pull-secret" not registered Jul 10 02:07:51.065158 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.035251 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29144f4e-ebaf-40b4-b9f1-9ee1fa371890-var-lib-kubelet\") pod \"tuned-68dsx\" (UID: \"29144f4e-ebaf-40b4-b9f1-9ee1fa371890\") " pod="openshift-cluster-node-tuning-operator/tuned-68dsx" Jul 10 02:07:51.065158 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.035276 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-host-run-ovn-kubernetes\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:51.065158 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.035315 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-node-log\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:51.065158 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.035337 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-host-cni-bin\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:51.065158 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.035370 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fad9e446-daca-42e4-a618-5defb2494fc3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-f9csf\" (UID: \"fad9e446-daca-42e4-a618-5defb2494fc3\") " pod="openshift-multus/multus-additional-cni-plugins-f9csf" Jul 10 02:07:51.065158 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.035376 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-host-cni-bin\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:51.065158 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.035398 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0fe32bca-a055-4658-b899-d22fbd705e41-etc-selinux\") pod \"aws-ebs-csi-driver-node-vtbg2\" (UID: \"0fe32bca-a055-4658-b899-d22fbd705e41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vtbg2" Jul 10 02:07:51.065158 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.035445 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/452d0117-6775-4e71-acc6-4e2fc639e369-agent-certs\") pod \"konnectivity-agent-cprsf\" (UID: \"452d0117-6775-4e71-acc6-4e2fc639e369\") " pod="kube-system/konnectivity-agent-cprsf" Jul 10 02:07:51.065158 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.035472 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-run-ovn\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:51.065158 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.035497 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0fe32bca-a055-4658-b899-d22fbd705e41-etc-selinux\") pod \"aws-ebs-csi-driver-node-vtbg2\" (UID: \"0fe32bca-a055-4658-b899-d22fbd705e41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vtbg2" Jul 10 02:07:51.065158 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.035573 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-run-ovn\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:51.065158 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.035498 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/29144f4e-ebaf-40b4-b9f1-9ee1fa371890-etc-modprobe-d\") pod \"tuned-68dsx\" (UID: \"29144f4e-ebaf-40b4-b9f1-9ee1fa371890\") " pod="openshift-cluster-node-tuning-operator/tuned-68dsx" Jul 10 02:07:51.065158 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.035602 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/29144f4e-ebaf-40b4-b9f1-9ee1fa371890-etc-modprobe-d\") pod \"tuned-68dsx\" (UID: \"29144f4e-ebaf-40b4-b9f1-9ee1fa371890\") " pod="openshift-cluster-node-tuning-operator/tuned-68dsx" Jul 10 02:07:51.065864 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.035623 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fad9e446-daca-42e4-a618-5defb2494fc3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-f9csf\" (UID: \"fad9e446-daca-42e4-a618-5defb2494fc3\") " pod="openshift-multus/multus-additional-cni-plugins-f9csf" Jul 10 02:07:51.065864 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.035656 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-ovnkube-script-lib\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:51.065864 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.035715 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/29144f4e-ebaf-40b4-b9f1-9ee1fa371890-etc-systemd\") pod \"tuned-68dsx\" (UID: \"29144f4e-ebaf-40b4-b9f1-9ee1fa371890\") " pod="openshift-cluster-node-tuning-operator/tuned-68dsx" Jul 10 02:07:51.065864 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.035743 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fad9e446-daca-42e4-a618-5defb2494fc3-os-release\") pod \"multus-additional-cni-plugins-f9csf\" (UID: \"fad9e446-daca-42e4-a618-5defb2494fc3\") " pod="openshift-multus/multus-additional-cni-plugins-f9csf" Jul 10 02:07:51.065864 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.035769 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f9v2\" (UniqueName: \"kubernetes.io/projected/fad9e446-daca-42e4-a618-5defb2494fc3-kube-api-access-7f9v2\") pod \"multus-additional-cni-plugins-f9csf\" (UID: \"fad9e446-daca-42e4-a618-5defb2494fc3\") " pod="openshift-multus/multus-additional-cni-plugins-f9csf" Jul 10 02:07:51.065864 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.035797 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8ccffaf3-b29f-4f58-ba37-5ab6f4992d19-hosts-file\") pod \"node-resolver-xgndm\" (UID: \"8ccffaf3-b29f-4f58-ba37-5ab6f4992d19\") " pod="openshift-dns/node-resolver-xgndm" Jul 10 02:07:51.065864 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.035827 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0fe32bca-a055-4658-b899-d22fbd705e41-socket-dir\") pod \"aws-ebs-csi-driver-node-vtbg2\" (UID: \"0fe32bca-a055-4658-b899-d22fbd705e41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vtbg2" Jul 10 02:07:51.065864 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.035847 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/29144f4e-ebaf-40b4-b9f1-9ee1fa371890-etc-systemd\") pod \"tuned-68dsx\" (UID: \"29144f4e-ebaf-40b4-b9f1-9ee1fa371890\") " pod="openshift-cluster-node-tuning-operator/tuned-68dsx" Jul 10 02:07:51.065864 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.035853 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/54d67123-d94f-4ae2-8f94-ec576ba2e8b3-host-slash\") pod \"iptables-alerter-58txz\" (UID: \"54d67123-d94f-4ae2-8f94-ec576ba2e8b3\") " pod="openshift-network-operator/iptables-alerter-58txz" Jul 10 02:07:51.065864 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.035890 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/54d67123-d94f-4ae2-8f94-ec576ba2e8b3-host-slash\") pod \"iptables-alerter-58txz\" (UID: \"54d67123-d94f-4ae2-8f94-ec576ba2e8b3\") " pod="openshift-network-operator/iptables-alerter-58txz" Jul 10 02:07:51.065864 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.035898 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-systemd-units\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:51.065864 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.035925 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fad9e446-daca-42e4-a618-5defb2494fc3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-f9csf\" (UID: \"fad9e446-daca-42e4-a618-5defb2494fc3\") " pod="openshift-multus/multus-additional-cni-plugins-f9csf" Jul 10 02:07:51.065864 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.035928 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/86176d1c-3214-46c1-91fe-94f087e0bda4-host\") pod \"node-ca-8t46j\" (UID: \"86176d1c-3214-46c1-91fe-94f087e0bda4\") " pod="openshift-image-registry/node-ca-8t46j" Jul 10 02:07:51.065864 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.035970 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-ovnkube-config\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:51.068407 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.035981 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-systemd-units\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:51.068407 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.035975 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/29144f4e-ebaf-40b4-b9f1-9ee1fa371890-etc-sysctl-conf\") pod \"tuned-68dsx\" (UID: \"29144f4e-ebaf-40b4-b9f1-9ee1fa371890\") " pod="openshift-cluster-node-tuning-operator/tuned-68dsx" Jul 10 02:07:51.068407 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.036042 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fad9e446-daca-42e4-a618-5defb2494fc3-os-release\") pod \"multus-additional-cni-plugins-f9csf\" (UID: \"fad9e446-daca-42e4-a618-5defb2494fc3\") " pod="openshift-multus/multus-additional-cni-plugins-f9csf" Jul 10 02:07:51.068407 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.036042 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ccffaf3-b29f-4f58-ba37-5ab6f4992d19-tmp-dir\") pod \"node-resolver-xgndm\" (UID: \"8ccffaf3-b29f-4f58-ba37-5ab6f4992d19\") " pod="openshift-dns/node-resolver-xgndm" Jul 10 02:07:51.068407 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.036075 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-var-lib-openvswitch\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:51.068407 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.036083 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/86176d1c-3214-46c1-91fe-94f087e0bda4-host\") pod \"node-ca-8t46j\" (UID: \"86176d1c-3214-46c1-91fe-94f087e0bda4\") " pod="openshift-image-registry/node-ca-8t46j" Jul 10 02:07:51.068407 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.036107 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rl78\" (UniqueName: \"kubernetes.io/projected/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-kube-api-access-5rl78\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:51.068407 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.036100 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/29144f4e-ebaf-40b4-b9f1-9ee1fa371890-etc-sysctl-conf\") pod \"tuned-68dsx\" (UID: \"29144f4e-ebaf-40b4-b9f1-9ee1fa371890\") " pod="openshift-cluster-node-tuning-operator/tuned-68dsx" Jul 10 02:07:51.068407 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.036141 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/29144f4e-ebaf-40b4-b9f1-9ee1fa371890-host\") pod \"tuned-68dsx\" (UID: \"29144f4e-ebaf-40b4-b9f1-9ee1fa371890\") " pod="openshift-cluster-node-tuning-operator/tuned-68dsx" Jul 10 02:07:51.068407 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.036168 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/29144f4e-ebaf-40b4-b9f1-9ee1fa371890-etc-tuned\") pod \"tuned-68dsx\" (UID: \"29144f4e-ebaf-40b4-b9f1-9ee1fa371890\") " pod="openshift-cluster-node-tuning-operator/tuned-68dsx" Jul 10 02:07:51.068407 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.036205 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-ovnkube-script-lib\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:51.068407 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.036216 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fad9e446-daca-42e4-a618-5defb2494fc3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-f9csf\" (UID: \"fad9e446-daca-42e4-a618-5defb2494fc3\") " pod="openshift-multus/multus-additional-cni-plugins-f9csf" Jul 10 02:07:51.068407 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.036313 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29144f4e-ebaf-40b4-b9f1-9ee1fa371890-tmp\") pod \"tuned-68dsx\" (UID: \"29144f4e-ebaf-40b4-b9f1-9ee1fa371890\") " pod="openshift-cluster-node-tuning-operator/tuned-68dsx" Jul 10 02:07:51.068407 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.036321 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-var-lib-openvswitch\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:51.069544 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.036345 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fad9e446-daca-42e4-a618-5defb2494fc3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f9csf\" (UID: \"fad9e446-daca-42e4-a618-5defb2494fc3\") " pod="openshift-multus/multus-additional-cni-plugins-f9csf" Jul 10 02:07:51.069544 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.036373 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0fe32bca-a055-4658-b899-d22fbd705e41-socket-dir\") pod \"aws-ebs-csi-driver-node-vtbg2\" (UID: \"0fe32bca-a055-4658-b899-d22fbd705e41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vtbg2" Jul 10 02:07:51.069544 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.036381 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ccffaf3-b29f-4f58-ba37-5ab6f4992d19-tmp-dir\") pod \"node-resolver-xgndm\" (UID: \"8ccffaf3-b29f-4f58-ba37-5ab6f4992d19\") " pod="openshift-dns/node-resolver-xgndm" Jul 10 02:07:51.069544 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.036421 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/29144f4e-ebaf-40b4-b9f1-9ee1fa371890-host\") pod \"tuned-68dsx\" (UID: \"29144f4e-ebaf-40b4-b9f1-9ee1fa371890\") " pod="openshift-cluster-node-tuning-operator/tuned-68dsx" Jul 10 02:07:51.069544 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.036468 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8ccffaf3-b29f-4f58-ba37-5ab6f4992d19-hosts-file\") pod \"node-resolver-xgndm\" (UID: \"8ccffaf3-b29f-4f58-ba37-5ab6f4992d19\") " pod="openshift-dns/node-resolver-xgndm" Jul 10 02:07:51.069544 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.036500 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-host-run-netns\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:51.069544 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.036510 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fad9e446-daca-42e4-a618-5defb2494fc3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f9csf\" (UID: \"fad9e446-daca-42e4-a618-5defb2494fc3\") " pod="openshift-multus/multus-additional-cni-plugins-f9csf" Jul 10 02:07:51.069544 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.036535 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/29144f4e-ebaf-40b4-b9f1-9ee1fa371890-etc-sysconfig\") pod \"tuned-68dsx\" (UID: \"29144f4e-ebaf-40b4-b9f1-9ee1fa371890\") " pod="openshift-cluster-node-tuning-operator/tuned-68dsx" Jul 10 02:07:51.069544 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.036552 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-host-run-netns\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:51.069544 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.036589 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0fe32bca-a055-4658-b899-d22fbd705e41-device-dir\") pod \"aws-ebs-csi-driver-node-vtbg2\" (UID: \"0fe32bca-a055-4658-b899-d22fbd705e41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vtbg2" Jul 10 02:07:51.069544 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.036696 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/29144f4e-ebaf-40b4-b9f1-9ee1fa371890-etc-sysconfig\") pod \"tuned-68dsx\" (UID: \"29144f4e-ebaf-40b4-b9f1-9ee1fa371890\") " pod="openshift-cluster-node-tuning-operator/tuned-68dsx" Jul 10 02:07:51.069544 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.036734 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0fe32bca-a055-4658-b899-d22fbd705e41-device-dir\") pod \"aws-ebs-csi-driver-node-vtbg2\" (UID: \"0fe32bca-a055-4658-b899-d22fbd705e41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vtbg2" Jul 10 02:07:51.069544 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.038596 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/452d0117-6775-4e71-acc6-4e2fc639e369-agent-certs\") pod \"konnectivity-agent-cprsf\" (UID: \"452d0117-6775-4e71-acc6-4e2fc639e369\") " pod="kube-system/konnectivity-agent-cprsf" Jul 10 02:07:51.069544 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.038845 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29144f4e-ebaf-40b4-b9f1-9ee1fa371890-tmp\") pod \"tuned-68dsx\" (UID: \"29144f4e-ebaf-40b4-b9f1-9ee1fa371890\") " pod="openshift-cluster-node-tuning-operator/tuned-68dsx" Jul 10 02:07:51.069544 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.038977 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/29144f4e-ebaf-40b4-b9f1-9ee1fa371890-etc-tuned\") pod \"tuned-68dsx\" (UID: \"29144f4e-ebaf-40b4-b9f1-9ee1fa371890\") " pod="openshift-cluster-node-tuning-operator/tuned-68dsx" Jul 10 02:07:51.070951 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:51.046743 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 02:07:51.070951 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:51.046768 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 02:07:51.070951 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:51.046782 2566 projected.go:196] Error preparing data for projected volume kube-api-access-lfmw4 for pod openshift-network-diagnostics/network-check-target-ll898: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 02:07:51.070951 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:51.046862 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fea8615-df10-4ccb-84cc-8d382f7bdafe-kube-api-access-lfmw4 podName:2fea8615-df10-4ccb-84cc-8d382f7bdafe nodeName:}" failed. No retries permitted until 2026-07-10 02:07:51.546842692 +0000 UTC m=+3.123119777 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lfmw4" (UniqueName: "kubernetes.io/projected/2fea8615-df10-4ccb-84cc-8d382f7bdafe-kube-api-access-lfmw4") pod "network-check-target-ll898" (UID: "2fea8615-df10-4ccb-84cc-8d382f7bdafe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 02:07:51.070951 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.051429 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f9v2\" (UniqueName: \"kubernetes.io/projected/fad9e446-daca-42e4-a618-5defb2494fc3-kube-api-access-7f9v2\") pod \"multus-additional-cni-plugins-f9csf\" (UID: \"fad9e446-daca-42e4-a618-5defb2494fc3\") " pod="openshift-multus/multus-additional-cni-plugins-f9csf" Jul 10 02:07:51.070951 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.051599 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9g42\" (UniqueName: \"kubernetes.io/projected/54d67123-d94f-4ae2-8f94-ec576ba2e8b3-kube-api-access-q9g42\") pod \"iptables-alerter-58txz\" (UID: \"54d67123-d94f-4ae2-8f94-ec576ba2e8b3\") " pod="openshift-network-operator/iptables-alerter-58txz" Jul 10 02:07:51.070951 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.051965 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmb5f\" (UniqueName: \"kubernetes.io/projected/0fe32bca-a055-4658-b899-d22fbd705e41-kube-api-access-lmb5f\") pod \"aws-ebs-csi-driver-node-vtbg2\" (UID: \"0fe32bca-a055-4658-b899-d22fbd705e41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vtbg2" Jul 10 02:07:51.070951 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.052904 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rl78\" (UniqueName: \"kubernetes.io/projected/5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc-kube-api-access-5rl78\") pod \"ovnkube-node-d8d5l\" (UID: \"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:51.070951 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.053660 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjv5d\" (UniqueName: \"kubernetes.io/projected/29144f4e-ebaf-40b4-b9f1-9ee1fa371890-kube-api-access-tjv5d\") pod \"tuned-68dsx\" (UID: \"29144f4e-ebaf-40b4-b9f1-9ee1fa371890\") " pod="openshift-cluster-node-tuning-operator/tuned-68dsx" Jul 10 02:07:51.070951 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.053709 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpdwc\" (UniqueName: \"kubernetes.io/projected/86176d1c-3214-46c1-91fe-94f087e0bda4-kube-api-access-wpdwc\") pod \"node-ca-8t46j\" (UID: \"86176d1c-3214-46c1-91fe-94f087e0bda4\") " pod="openshift-image-registry/node-ca-8t46j" Jul 10 02:07:51.070951 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.054208 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl5f2\" (UniqueName: \"kubernetes.io/projected/8ccffaf3-b29f-4f58-ba37-5ab6f4992d19-kube-api-access-nl5f2\") pod \"node-resolver-xgndm\" (UID: \"8ccffaf3-b29f-4f58-ba37-5ab6f4992d19\") " pod="openshift-dns/node-resolver-xgndm" Jul 10 02:07:51.070951 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.055108 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpskb\" (UniqueName: \"kubernetes.io/projected/8bf086af-02b9-43f2-935f-474133fef4cf-kube-api-access-dpskb\") pod \"network-metrics-daemon-6hfgs\" (UID: \"8bf086af-02b9-43f2-935f-474133fef4cf\") " pod="openshift-multus/network-metrics-daemon-6hfgs" Jul 10 02:07:51.174061 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.106644 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wpxd6" Jul 10 02:07:51.174061 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.116609 2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 02:07:51.174061 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.116776 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cprsf" Jul 10 02:07:51.174061 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.128677 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:07:51.174061 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:51.129371 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod452d0117_6775_4e71_acc6_4e2fc639e369.slice/crio-c59de7eb4a24699670f03bbeead7138ab14d23509e37b0445ecc312b2441d93f WatchSource:0}: Error finding container c59de7eb4a24699670f03bbeead7138ab14d23509e37b0445ecc312b2441d93f: Status 404 returned error can't find the container with id c59de7eb4a24699670f03bbeead7138ab14d23509e37b0445ecc312b2441d93f Jul 10 02:07:51.174061 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.135952 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xgndm" Jul 10 02:07:51.174061 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:51.144551 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cf43a6d_60e3_4c7d_9d72_ecb5a0a68afc.slice/crio-515371af27f7bdc6b18d45f4bc04dc0e6818b08d2a08cb91f4d2b21e4f3c8c4c WatchSource:0}: Error finding container 515371af27f7bdc6b18d45f4bc04dc0e6818b08d2a08cb91f4d2b21e4f3c8c4c: Status 404 returned error can't find the container with id 515371af27f7bdc6b18d45f4bc04dc0e6818b08d2a08cb91f4d2b21e4f3c8c4c Jul 10 02:07:51.174061 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:51.148998 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ccffaf3_b29f_4f58_ba37_5ab6f4992d19.slice/crio-b2a81be7d2c01b0903ca01162963b81628bcd6647f9118de55161f030c446a48 WatchSource:0}: Error finding container b2a81be7d2c01b0903ca01162963b81628bcd6647f9118de55161f030c446a48: Status 404 returned error can't find the container with id b2a81be7d2c01b0903ca01162963b81628bcd6647f9118de55161f030c446a48 Jul 10 02:07:51.174061 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.149843 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vtbg2" Jul 10 02:07:51.174061 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.157934 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-68dsx" Jul 10 02:07:51.174061 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:51.171438 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29144f4e_ebaf_40b4_b9f1_9ee1fa371890.slice/crio-923d2b2fd43a8798cf014e2fb6e57bf1c459bf9008259eb7db6130fb5d9b927a WatchSource:0}: Error finding container 923d2b2fd43a8798cf014e2fb6e57bf1c459bf9008259eb7db6130fb5d9b927a: Status 404 returned error can't find the container with id 923d2b2fd43a8798cf014e2fb6e57bf1c459bf9008259eb7db6130fb5d9b927a Jul 10 02:07:51.175711 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.175428 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8t46j" Jul 10 02:07:51.247648 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.218346 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-f9csf" Jul 10 02:07:51.247648 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.230304 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-58txz" Jul 10 02:07:51.348223 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:51.279640 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86176d1c_3214_46c1_91fe_94f087e0bda4.slice/crio-ed52c2e3243a2e912b9fb08136d6d6202b83f40d54deb1c5e56962585cda15ff WatchSource:0}: Error finding container ed52c2e3243a2e912b9fb08136d6d6202b83f40d54deb1c5e56962585cda15ff: Status 404 returned error can't find the container with id ed52c2e3243a2e912b9fb08136d6d6202b83f40d54deb1c5e56962585cda15ff Jul 10 02:07:51.348223 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:51.284435 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54d67123_d94f_4ae2_8f94_ec576ba2e8b3.slice/crio-e899720674436280ddd84336eaf9e430fd5c796adf5e09b54c54c253aba626b9 WatchSource:0}: Error finding container e899720674436280ddd84336eaf9e430fd5c796adf5e09b54c54c253aba626b9: Status 404 returned error can't find the container with id e899720674436280ddd84336eaf9e430fd5c796adf5e09b54c54c253aba626b9 Jul 10 02:07:51.348223 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:07:51.288873 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfad9e446_daca_42e4_a618_5defb2494fc3.slice/crio-d5560aabd13094a40f6e0633b5bbd08e217a4086d2e5da5614cb9ad69db99d7f WatchSource:0}: Error finding container d5560aabd13094a40f6e0633b5bbd08e217a4086d2e5da5614cb9ad69db99d7f: Status 404 returned error can't find the container with id d5560aabd13094a40f6e0633b5bbd08e217a4086d2e5da5614cb9ad69db99d7f Jul 10 02:07:51.570344 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.540910 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/709370a6-295c-4330-bfa5-50ca602a28fb-original-pull-secret\") pod \"global-pull-secret-syncer-pcfzz\" (UID: \"709370a6-295c-4330-bfa5-50ca602a28fb\") " pod="kube-system/global-pull-secret-syncer-pcfzz" Jul 10 02:07:51.570344 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.541097 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bf086af-02b9-43f2-935f-474133fef4cf-metrics-certs\") pod \"network-metrics-daemon-6hfgs\" (UID: \"8bf086af-02b9-43f2-935f-474133fef4cf\") " pod="openshift-multus/network-metrics-daemon-6hfgs" Jul 10 02:07:51.570344 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:51.541101 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 02:07:51.570344 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:51.541177 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/709370a6-295c-4330-bfa5-50ca602a28fb-original-pull-secret podName:709370a6-295c-4330-bfa5-50ca602a28fb nodeName:}" failed. No retries permitted until 2026-07-10 02:07:52.541154514 +0000 UTC m=+4.117431584 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/709370a6-295c-4330-bfa5-50ca602a28fb-original-pull-secret") pod "global-pull-secret-syncer-pcfzz" (UID: "709370a6-295c-4330-bfa5-50ca602a28fb") : object "kube-system"/"original-pull-secret" not registered Jul 10 02:07:51.570344 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:51.541208 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 02:07:51.570344 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:51.541263 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bf086af-02b9-43f2-935f-474133fef4cf-metrics-certs podName:8bf086af-02b9-43f2-935f-474133fef4cf nodeName:}" failed. No retries permitted until 2026-07-10 02:07:52.541248566 +0000 UTC m=+4.117525641 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8bf086af-02b9-43f2-935f-474133fef4cf-metrics-certs") pod "network-metrics-daemon-6hfgs" (UID: "8bf086af-02b9-43f2-935f-474133fef4cf") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 02:07:51.657319 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.642362 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfmw4\" (UniqueName: \"kubernetes.io/projected/2fea8615-df10-4ccb-84cc-8d382f7bdafe-kube-api-access-lfmw4\") pod \"network-check-target-ll898\" (UID: \"2fea8615-df10-4ccb-84cc-8d382f7bdafe\") " pod="openshift-network-diagnostics/network-check-target-ll898" Jul 10 02:07:51.657319 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:51.642563 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 02:07:51.657319 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:51.642591 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 02:07:51.657319 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:51.642603 2566 projected.go:196] Error preparing data for projected volume kube-api-access-lfmw4 for pod openshift-network-diagnostics/network-check-target-ll898: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 02:07:51.657319 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:51.642678 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fea8615-df10-4ccb-84cc-8d382f7bdafe-kube-api-access-lfmw4 podName:2fea8615-df10-4ccb-84cc-8d382f7bdafe nodeName:}" failed. No retries permitted until 2026-07-10 02:07:52.642662671 +0000 UTC m=+4.218939737 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lfmw4" (UniqueName: "kubernetes.io/projected/2fea8615-df10-4ccb-84cc-8d382f7bdafe-kube-api-access-lfmw4") pod "network-check-target-ll898" (UID: "2fea8615-df10-4ccb-84cc-8d382f7bdafe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 02:07:51.855760 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.855618 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 02:02:51 +0000 UTC" deadline="2028-02-25 03:43:43.078627468 +0000 UTC" Jul 10 02:07:51.855760 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:51.855692 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14281h35m51.222940842s" Jul 10 02:07:52.005089 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:52.004992 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcfzz" Jul 10 02:07:52.005260 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:52.005152 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcfzz" podUID="709370a6-295c-4330-bfa5-50ca602a28fb" Jul 10 02:07:52.016974 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:52.016901 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cprsf" event={"ID":"452d0117-6775-4e71-acc6-4e2fc639e369","Type":"ContainerStarted","Data":"c59de7eb4a24699670f03bbeead7138ab14d23509e37b0445ecc312b2441d93f"} Jul 10 02:07:52.018557 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:52.018524 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f9csf" event={"ID":"fad9e446-daca-42e4-a618-5defb2494fc3","Type":"ContainerStarted","Data":"d5560aabd13094a40f6e0633b5bbd08e217a4086d2e5da5614cb9ad69db99d7f"} Jul 10 02:07:52.019892 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:52.019852 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8t46j" event={"ID":"86176d1c-3214-46c1-91fe-94f087e0bda4","Type":"ContainerStarted","Data":"ed52c2e3243a2e912b9fb08136d6d6202b83f40d54deb1c5e56962585cda15ff"} Jul 10 02:07:52.021170 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:52.021144 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-68dsx" event={"ID":"29144f4e-ebaf-40b4-b9f1-9ee1fa371890","Type":"ContainerStarted","Data":"923d2b2fd43a8798cf014e2fb6e57bf1c459bf9008259eb7db6130fb5d9b927a"} Jul 10 02:07:52.022359 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:52.022336 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vtbg2" event={"ID":"0fe32bca-a055-4658-b899-d22fbd705e41","Type":"ContainerStarted","Data":"51f10490613b49383c093ccf1bb8708d4e309d1a1a927595395f218c21279767"} Jul 10 02:07:52.023798 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:52.023750 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xgndm" event={"ID":"8ccffaf3-b29f-4f58-ba37-5ab6f4992d19","Type":"ContainerStarted","Data":"b2a81be7d2c01b0903ca01162963b81628bcd6647f9118de55161f030c446a48"} Jul 10 02:07:52.025027 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:52.024998 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" event={"ID":"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc","Type":"ContainerStarted","Data":"515371af27f7bdc6b18d45f4bc04dc0e6818b08d2a08cb91f4d2b21e4f3c8c4c"} Jul 10 02:07:52.026391 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:52.026365 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wpxd6" event={"ID":"a49e0328-71c1-43ce-80e5-83d0c0862cf9","Type":"ContainerStarted","Data":"a843e21c1727711a7eb5f47f0c8a8cd91cacf29ef080d8703ef7dc3a6ee1c802"} Jul 10 02:07:52.027651 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:52.027624 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-58txz" event={"ID":"54d67123-d94f-4ae2-8f94-ec576ba2e8b3","Type":"ContainerStarted","Data":"e899720674436280ddd84336eaf9e430fd5c796adf5e09b54c54c253aba626b9"} Jul 10 02:07:52.550152 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:52.549845 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bf086af-02b9-43f2-935f-474133fef4cf-metrics-certs\") pod \"network-metrics-daemon-6hfgs\" (UID: \"8bf086af-02b9-43f2-935f-474133fef4cf\") " pod="openshift-multus/network-metrics-daemon-6hfgs" Jul 10 02:07:52.550152 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:52.549911 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/709370a6-295c-4330-bfa5-50ca602a28fb-original-pull-secret\") pod \"global-pull-secret-syncer-pcfzz\" (UID: \"709370a6-295c-4330-bfa5-50ca602a28fb\") " pod="kube-system/global-pull-secret-syncer-pcfzz" Jul 10 02:07:52.550152 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:52.550014 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 02:07:52.550152 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:52.550086 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bf086af-02b9-43f2-935f-474133fef4cf-metrics-certs podName:8bf086af-02b9-43f2-935f-474133fef4cf nodeName:}" failed. No retries permitted until 2026-07-10 02:07:54.550066066 +0000 UTC m=+6.126343150 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8bf086af-02b9-43f2-935f-474133fef4cf-metrics-certs") pod "network-metrics-daemon-6hfgs" (UID: "8bf086af-02b9-43f2-935f-474133fef4cf") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 02:07:52.550808 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:52.550020 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 02:07:52.550808 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:52.550497 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/709370a6-295c-4330-bfa5-50ca602a28fb-original-pull-secret podName:709370a6-295c-4330-bfa5-50ca602a28fb nodeName:}" failed. No retries permitted until 2026-07-10 02:07:54.550479456 +0000 UTC m=+6.126756525 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/709370a6-295c-4330-bfa5-50ca602a28fb-original-pull-secret") pod "global-pull-secret-syncer-pcfzz" (UID: "709370a6-295c-4330-bfa5-50ca602a28fb") : object "kube-system"/"original-pull-secret" not registered Jul 10 02:07:52.651114 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:52.651069 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfmw4\" (UniqueName: \"kubernetes.io/projected/2fea8615-df10-4ccb-84cc-8d382f7bdafe-kube-api-access-lfmw4\") pod \"network-check-target-ll898\" (UID: \"2fea8615-df10-4ccb-84cc-8d382f7bdafe\") " pod="openshift-network-diagnostics/network-check-target-ll898" Jul 10 02:07:52.651331 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:52.651313 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 02:07:52.651397 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:52.651342 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 02:07:52.651397 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:52.651357 2566 projected.go:196] Error preparing data for projected volume kube-api-access-lfmw4 for pod openshift-network-diagnostics/network-check-target-ll898: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 02:07:52.651505 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:52.651425 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fea8615-df10-4ccb-84cc-8d382f7bdafe-kube-api-access-lfmw4 podName:2fea8615-df10-4ccb-84cc-8d382f7bdafe nodeName:}" failed. No retries permitted until 2026-07-10 02:07:54.65140427 +0000 UTC m=+6.227681354 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lfmw4" (UniqueName: "kubernetes.io/projected/2fea8615-df10-4ccb-84cc-8d382f7bdafe-kube-api-access-lfmw4") pod "network-check-target-ll898" (UID: "2fea8615-df10-4ccb-84cc-8d382f7bdafe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 02:07:52.856151 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:52.856046 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 02:02:51 +0000 UTC" deadline="2028-01-23 15:18:18.211306496 +0000 UTC" Jul 10 02:07:52.856151 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:52.856095 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13501h10m25.355216514s" Jul 10 02:07:53.018014 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:53.017943 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hfgs" Jul 10 02:07:53.018214 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:53.017985 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ll898" Jul 10 02:07:53.018214 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:53.018132 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6hfgs" podUID="8bf086af-02b9-43f2-935f-474133fef4cf" Jul 10 02:07:53.018337 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:53.018207 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ll898" podUID="2fea8615-df10-4ccb-84cc-8d382f7bdafe" Jul 10 02:07:54.005327 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:54.005288 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcfzz" Jul 10 02:07:54.005907 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:54.005440 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcfzz" podUID="709370a6-295c-4330-bfa5-50ca602a28fb" Jul 10 02:07:54.568706 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:54.567846 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bf086af-02b9-43f2-935f-474133fef4cf-metrics-certs\") pod \"network-metrics-daemon-6hfgs\" (UID: \"8bf086af-02b9-43f2-935f-474133fef4cf\") " pod="openshift-multus/network-metrics-daemon-6hfgs" Jul 10 02:07:54.568706 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:54.567915 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/709370a6-295c-4330-bfa5-50ca602a28fb-original-pull-secret\") pod \"global-pull-secret-syncer-pcfzz\" (UID: \"709370a6-295c-4330-bfa5-50ca602a28fb\") " pod="kube-system/global-pull-secret-syncer-pcfzz" Jul 10 02:07:54.568706 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:54.568068 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 02:07:54.568706 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:54.568137 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/709370a6-295c-4330-bfa5-50ca602a28fb-original-pull-secret podName:709370a6-295c-4330-bfa5-50ca602a28fb nodeName:}" failed. No retries permitted until 2026-07-10 02:07:58.568117601 +0000 UTC m=+10.144394676 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/709370a6-295c-4330-bfa5-50ca602a28fb-original-pull-secret") pod "global-pull-secret-syncer-pcfzz" (UID: "709370a6-295c-4330-bfa5-50ca602a28fb") : object "kube-system"/"original-pull-secret" not registered Jul 10 02:07:54.568706 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:54.568595 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 02:07:54.568706 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:54.568651 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bf086af-02b9-43f2-935f-474133fef4cf-metrics-certs podName:8bf086af-02b9-43f2-935f-474133fef4cf nodeName:}" failed. No retries permitted until 2026-07-10 02:07:58.568633411 +0000 UTC m=+10.144910492 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8bf086af-02b9-43f2-935f-474133fef4cf-metrics-certs") pod "network-metrics-daemon-6hfgs" (UID: "8bf086af-02b9-43f2-935f-474133fef4cf") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 02:07:54.669387 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:54.668660 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfmw4\" (UniqueName: \"kubernetes.io/projected/2fea8615-df10-4ccb-84cc-8d382f7bdafe-kube-api-access-lfmw4\") pod \"network-check-target-ll898\" (UID: \"2fea8615-df10-4ccb-84cc-8d382f7bdafe\") " pod="openshift-network-diagnostics/network-check-target-ll898" Jul 10 02:07:54.669387 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:54.668899 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 02:07:54.669387 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:54.668920 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 02:07:54.669387 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:54.668935 2566 projected.go:196] Error preparing data for projected volume kube-api-access-lfmw4 for pod openshift-network-diagnostics/network-check-target-ll898: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 02:07:54.669387 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:54.669000 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fea8615-df10-4ccb-84cc-8d382f7bdafe-kube-api-access-lfmw4 podName:2fea8615-df10-4ccb-84cc-8d382f7bdafe nodeName:}" failed. No retries permitted until 2026-07-10 02:07:58.668980056 +0000 UTC m=+10.245257126 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lfmw4" (UniqueName: "kubernetes.io/projected/2fea8615-df10-4ccb-84cc-8d382f7bdafe-kube-api-access-lfmw4") pod "network-check-target-ll898" (UID: "2fea8615-df10-4ccb-84cc-8d382f7bdafe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 02:07:55.007702 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:55.005846 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hfgs" Jul 10 02:07:55.007702 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:55.006005 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6hfgs" podUID="8bf086af-02b9-43f2-935f-474133fef4cf" Jul 10 02:07:55.007702 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:55.006452 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ll898" Jul 10 02:07:55.007702 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:55.006576 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ll898" podUID="2fea8615-df10-4ccb-84cc-8d382f7bdafe" Jul 10 02:07:56.005365 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:56.005304 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcfzz" Jul 10 02:07:56.005565 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:56.005452 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcfzz" podUID="709370a6-295c-4330-bfa5-50ca602a28fb" Jul 10 02:07:57.006362 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:57.005792 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hfgs" Jul 10 02:07:57.006362 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:57.005842 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ll898" Jul 10 02:07:57.006362 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:57.005941 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6hfgs" podUID="8bf086af-02b9-43f2-935f-474133fef4cf" Jul 10 02:07:57.006362 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:57.006016 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ll898" podUID="2fea8615-df10-4ccb-84cc-8d382f7bdafe" Jul 10 02:07:58.004838 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:58.004801 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcfzz" Jul 10 02:07:58.005007 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:58.004946 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcfzz" podUID="709370a6-295c-4330-bfa5-50ca602a28fb" Jul 10 02:07:58.598483 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:58.598449 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bf086af-02b9-43f2-935f-474133fef4cf-metrics-certs\") pod \"network-metrics-daemon-6hfgs\" (UID: \"8bf086af-02b9-43f2-935f-474133fef4cf\") " pod="openshift-multus/network-metrics-daemon-6hfgs" Jul 10 02:07:58.598483 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:58.598493 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/709370a6-295c-4330-bfa5-50ca602a28fb-original-pull-secret\") pod \"global-pull-secret-syncer-pcfzz\" (UID: \"709370a6-295c-4330-bfa5-50ca602a28fb\") " pod="kube-system/global-pull-secret-syncer-pcfzz" Jul 10 02:07:58.598992 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:58.598642 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 02:07:58.598992 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:58.598645 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 02:07:58.598992 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:58.598709 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/709370a6-295c-4330-bfa5-50ca602a28fb-original-pull-secret podName:709370a6-295c-4330-bfa5-50ca602a28fb nodeName:}" failed. No retries permitted until 2026-07-10 02:08:06.598690506 +0000 UTC m=+18.174967573 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/709370a6-295c-4330-bfa5-50ca602a28fb-original-pull-secret") pod "global-pull-secret-syncer-pcfzz" (UID: "709370a6-295c-4330-bfa5-50ca602a28fb") : object "kube-system"/"original-pull-secret" not registered Jul 10 02:07:58.598992 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:58.598732 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bf086af-02b9-43f2-935f-474133fef4cf-metrics-certs podName:8bf086af-02b9-43f2-935f-474133fef4cf nodeName:}" failed. No retries permitted until 2026-07-10 02:08:06.598720942 +0000 UTC m=+18.174998005 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8bf086af-02b9-43f2-935f-474133fef4cf-metrics-certs") pod "network-metrics-daemon-6hfgs" (UID: "8bf086af-02b9-43f2-935f-474133fef4cf") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 02:07:58.698916 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:58.698878 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfmw4\" (UniqueName: \"kubernetes.io/projected/2fea8615-df10-4ccb-84cc-8d382f7bdafe-kube-api-access-lfmw4\") pod \"network-check-target-ll898\" (UID: \"2fea8615-df10-4ccb-84cc-8d382f7bdafe\") " pod="openshift-network-diagnostics/network-check-target-ll898" Jul 10 02:07:58.699122 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:58.699097 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 02:07:58.699122 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:58.699121 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 02:07:58.699331 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:58.699160 2566 projected.go:196] Error preparing data for projected volume kube-api-access-lfmw4 for pod openshift-network-diagnostics/network-check-target-ll898: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 02:07:58.699331 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:58.699218 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fea8615-df10-4ccb-84cc-8d382f7bdafe-kube-api-access-lfmw4 podName:2fea8615-df10-4ccb-84cc-8d382f7bdafe nodeName:}" failed. No retries permitted until 2026-07-10 02:08:06.699204606 +0000 UTC m=+18.275481670 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lfmw4" (UniqueName: "kubernetes.io/projected/2fea8615-df10-4ccb-84cc-8d382f7bdafe-kube-api-access-lfmw4") pod "network-check-target-ll898" (UID: "2fea8615-df10-4ccb-84cc-8d382f7bdafe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 02:07:59.006560 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:59.006472 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hfgs" Jul 10 02:07:59.006733 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:59.006610 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6hfgs" podUID="8bf086af-02b9-43f2-935f-474133fef4cf" Jul 10 02:07:59.006733 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:07:59.006699 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ll898" Jul 10 02:07:59.006868 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:07:59.006768 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ll898" podUID="2fea8615-df10-4ccb-84cc-8d382f7bdafe" Jul 10 02:08:00.005867 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:00.005825 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcfzz" Jul 10 02:08:00.006351 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:00.005967 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcfzz" podUID="709370a6-295c-4330-bfa5-50ca602a28fb" Jul 10 02:08:01.006027 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:01.005914 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ll898" Jul 10 02:08:01.006424 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:01.006066 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ll898" podUID="2fea8615-df10-4ccb-84cc-8d382f7bdafe" Jul 10 02:08:01.006424 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:01.006121 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hfgs" Jul 10 02:08:01.006424 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:01.006217 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6hfgs" podUID="8bf086af-02b9-43f2-935f-474133fef4cf" Jul 10 02:08:02.005240 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:02.004986 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcfzz" Jul 10 02:08:02.005434 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:02.005356 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcfzz" podUID="709370a6-295c-4330-bfa5-50ca602a28fb" Jul 10 02:08:02.055410 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:02.055369 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-68dsx" event={"ID":"29144f4e-ebaf-40b4-b9f1-9ee1fa371890","Type":"ContainerStarted","Data":"60923f6135a5b386a69aa2d70b3ab38fb8d855d826c1ae1c2c67abe1a682fd36"} Jul 10 02:08:02.056983 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:02.056954 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-65.ec2.internal" event={"ID":"305afbb58c749f5158dc47cb0c52154f","Type":"ContainerStarted","Data":"606eb2df804c0969fec58b85688811c97eee691f5604d8379948cf0a25a6506b"} Jul 10 02:08:02.091352 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:02.091281 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-68dsx" podStartSLOduration=2.7032533770000002 podStartE2EDuration="13.091258668s" totalImagesPullingTime="10.388005291s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 02:07:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:08:02.074766377 +0000 UTC m=+13.651043464" watchObservedRunningTime="2026-07-10 02:08:02.091258668 +0000 UTC m=+13.667535757" Jul 10 02:08:03.005445 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:03.005407 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ll898" Jul 10 02:08:03.005649 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:03.005539 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ll898" podUID="2fea8615-df10-4ccb-84cc-8d382f7bdafe" Jul 10 02:08:03.005649 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:03.005591 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hfgs" Jul 10 02:08:03.005769 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:03.005726 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6hfgs" podUID="8bf086af-02b9-43f2-935f-474133fef4cf" Jul 10 02:08:03.061265 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:03.061224 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vtbg2" event={"ID":"0fe32bca-a055-4658-b899-d22fbd705e41","Type":"ContainerStarted","Data":"f119f9c0378c0c611b7ec3c9f61b2b3c8e470605ac00e621b9f88d69251b925c"} Jul 10 02:08:03.062878 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:03.062850 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xgndm" event={"ID":"8ccffaf3-b29f-4f58-ba37-5ab6f4992d19","Type":"ContainerStarted","Data":"803c1eb0758a915f45b8d8e02a9efb6022be6864cde638acc65925d7f627eb03"} Jul 10 02:08:03.064436 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:03.064409 2566 generic.go:356] "Generic (PLEG): container finished" podID="aff7f4ed066dfa82229540d0126815cc" containerID="3d69896e5625d3c2d13042f9548bf1939496427d6499438f9e9034ca8cd4e084" exitCode=0 Jul 10 02:08:03.064574 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:03.064498 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-65.ec2.internal" event={"ID":"aff7f4ed066dfa82229540d0126815cc","Type":"ContainerDied","Data":"3d69896e5625d3c2d13042f9548bf1939496427d6499438f9e9034ca8cd4e084"} Jul 10 02:08:03.066265 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:03.066238 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cprsf" event={"ID":"452d0117-6775-4e71-acc6-4e2fc639e369","Type":"ContainerStarted","Data":"c57bf86979e16f182a9c0eda76f25821322baa4a367ab9a06f73345a044a691a"} Jul 10 02:08:03.067985 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:03.067959 2566 generic.go:356] "Generic (PLEG): container finished" podID="fad9e446-daca-42e4-a618-5defb2494fc3" containerID="54a0c0208aba5274a6ebde489015fb62aad55daedf65735373e9a89921159d74" exitCode=0 Jul 10 02:08:03.068119 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:03.068098 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f9csf" event={"ID":"fad9e446-daca-42e4-a618-5defb2494fc3","Type":"ContainerDied","Data":"54a0c0208aba5274a6ebde489015fb62aad55daedf65735373e9a89921159d74"} Jul 10 02:08:03.070353 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:03.070319 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8t46j" event={"ID":"86176d1c-3214-46c1-91fe-94f087e0bda4","Type":"ContainerStarted","Data":"2704759aba1b80bf5061d1f5a1f46e79757f125955fbd1df0e11400a09fea94c"} Jul 10 02:08:03.086791 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:03.086743 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-65.ec2.internal" podStartSLOduration=13.086721409999999 podStartE2EDuration="13.08672141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 02:07:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:08:02.091172855 +0000 UTC m=+13.667449942" watchObservedRunningTime="2026-07-10 02:08:03.08672141 +0000 UTC m=+14.662998497" Jul 10 02:08:03.123212 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:03.122479 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xgndm" podStartSLOduration=3.765374725 podStartE2EDuration="14.122455951s" totalImagesPullingTime="10.357081226s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 02:07:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:08:03.088092764 +0000 UTC m=+14.664369849" watchObservedRunningTime="2026-07-10 02:08:03.122455951 +0000 UTC m=+14.698733039" Jul 10 02:08:03.159306 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:03.159259 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-cprsf" podStartSLOduration=3.790478395 podStartE2EDuration="14.159240577s" totalImagesPullingTime="10.368762182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:07:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:08:03.15882692 +0000 UTC m=+14.735104008" watchObservedRunningTime="2026-07-10 02:08:03.159240577 +0000 UTC m=+14.735517663" Jul 10 02:08:03.175680 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:03.175636 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8t46j" podStartSLOduration=3.9532790330000003 podStartE2EDuration="14.175620654s" totalImagesPullingTime="10.222341621s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 02:07:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:08:03.175038806 +0000 UTC m=+14.751315892" watchObservedRunningTime="2026-07-10 02:08:03.175620654 +0000 UTC m=+14.751897739" Jul 10 02:08:04.005806 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:04.005769 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcfzz" Jul 10 02:08:04.005971 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:04.005914 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcfzz" podUID="709370a6-295c-4330-bfa5-50ca602a28fb" Jul 10 02:08:04.074029 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:04.073987 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-65.ec2.internal" event={"ID":"aff7f4ed066dfa82229540d0126815cc","Type":"ContainerStarted","Data":"cbd2321620178a727b83560b59fba2b2737b875e77bc8656590c810053b8843d"} Jul 10 02:08:04.076198 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:04.076164 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-58txz" event={"ID":"54d67123-d94f-4ae2-8f94-ec576ba2e8b3","Type":"ContainerStarted","Data":"2817099f6bc2245dc4b9bd7b01af3464e4a1f4a3a8e84b0457735a8dd2e75b4d"} Jul 10 02:08:04.097624 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:04.096664 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-65.ec2.internal" podStartSLOduration=14.096642487 podStartE2EDuration="14.096642487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 02:07:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:08:04.096065887 +0000 UTC m=+15.672342974" watchObservedRunningTime="2026-07-10 02:08:04.096642487 +0000 UTC m=+15.672919573" Jul 10 02:08:04.271531 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:04.271441 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:08:04.363768 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:04.363708 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:08:04.601308 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:04.601223 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:08:04.665362 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:04.665323 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:08:04.725436 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:04.725375 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:08:05.005943 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:05.005833 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ll898" Jul 10 02:08:05.006122 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:05.005970 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ll898" podUID="2fea8615-df10-4ccb-84cc-8d382f7bdafe" Jul 10 02:08:05.006363 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:05.006343 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hfgs" Jul 10 02:08:05.006495 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:05.006470 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6hfgs" podUID="8bf086af-02b9-43f2-935f-474133fef4cf" Jul 10 02:08:06.005200 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:06.005166 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcfzz" Jul 10 02:08:06.005952 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:06.005290 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcfzz" podUID="709370a6-295c-4330-bfa5-50ca602a28fb" Jul 10 02:08:06.251060 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:06.251018 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-cprsf" Jul 10 02:08:06.251840 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:06.251816 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-cprsf" Jul 10 02:08:06.269242 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:06.269156 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-58txz" podStartSLOduration=7.057333513 podStartE2EDuration="17.269136761s" totalImagesPullingTime="10.211803248s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 02:07:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:08:04.120627272 +0000 UTC m=+15.696904359" watchObservedRunningTime="2026-07-10 02:08:06.269136761 +0000 UTC m=+17.845413848" Jul 10 02:08:06.661755 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:06.661679 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/709370a6-295c-4330-bfa5-50ca602a28fb-original-pull-secret\") pod \"global-pull-secret-syncer-pcfzz\" (UID: \"709370a6-295c-4330-bfa5-50ca602a28fb\") " pod="kube-system/global-pull-secret-syncer-pcfzz" Jul 10 02:08:06.661755 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:06.661755 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bf086af-02b9-43f2-935f-474133fef4cf-metrics-certs\") pod \"network-metrics-daemon-6hfgs\" (UID: \"8bf086af-02b9-43f2-935f-474133fef4cf\") " pod="openshift-multus/network-metrics-daemon-6hfgs" Jul 10 02:08:06.661967 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:06.661826 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 02:08:06.661967 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:06.661839 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 02:08:06.661967 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:06.661897 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bf086af-02b9-43f2-935f-474133fef4cf-metrics-certs podName:8bf086af-02b9-43f2-935f-474133fef4cf nodeName:}" failed. No retries permitted until 2026-07-10 02:08:22.66187799 +0000 UTC m=+34.238155055 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8bf086af-02b9-43f2-935f-474133fef4cf-metrics-certs") pod "network-metrics-daemon-6hfgs" (UID: "8bf086af-02b9-43f2-935f-474133fef4cf") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 02:08:06.661967 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:06.661914 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/709370a6-295c-4330-bfa5-50ca602a28fb-original-pull-secret podName:709370a6-295c-4330-bfa5-50ca602a28fb nodeName:}" failed. No retries permitted until 2026-07-10 02:08:22.661905533 +0000 UTC m=+34.238182612 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/709370a6-295c-4330-bfa5-50ca602a28fb-original-pull-secret") pod "global-pull-secret-syncer-pcfzz" (UID: "709370a6-295c-4330-bfa5-50ca602a28fb") : object "kube-system"/"original-pull-secret" not registered Jul 10 02:08:06.763036 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:06.763001 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfmw4\" (UniqueName: \"kubernetes.io/projected/2fea8615-df10-4ccb-84cc-8d382f7bdafe-kube-api-access-lfmw4\") pod \"network-check-target-ll898\" (UID: \"2fea8615-df10-4ccb-84cc-8d382f7bdafe\") " pod="openshift-network-diagnostics/network-check-target-ll898" Jul 10 02:08:06.763214 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:06.763186 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 02:08:06.763214 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:06.763207 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 02:08:06.763214 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:06.763217 2566 projected.go:196] Error preparing data for projected volume kube-api-access-lfmw4 for pod openshift-network-diagnostics/network-check-target-ll898: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 02:08:06.763331 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:06.763278 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fea8615-df10-4ccb-84cc-8d382f7bdafe-kube-api-access-lfmw4 podName:2fea8615-df10-4ccb-84cc-8d382f7bdafe nodeName:}" failed. No retries permitted until 2026-07-10 02:08:22.763262248 +0000 UTC m=+34.339539335 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lfmw4" (UniqueName: "kubernetes.io/projected/2fea8615-df10-4ccb-84cc-8d382f7bdafe-kube-api-access-lfmw4") pod "network-check-target-ll898" (UID: "2fea8615-df10-4ccb-84cc-8d382f7bdafe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 02:08:07.005641 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:07.005542 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ll898" Jul 10 02:08:07.006221 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:07.005702 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ll898" podUID="2fea8615-df10-4ccb-84cc-8d382f7bdafe" Jul 10 02:08:07.006221 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:07.005745 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hfgs" Jul 10 02:08:07.006221 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:07.005868 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6hfgs" podUID="8bf086af-02b9-43f2-935f-474133fef4cf" Jul 10 02:08:07.083066 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:07.083038 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-cprsf" Jul 10 02:08:07.083627 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:07.083585 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cprsf" Jul 10 02:08:07.155311 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:07.155267 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:08:08.004834 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:08.004800 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcfzz" Jul 10 02:08:08.004996 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:08.004918 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcfzz" podUID="709370a6-295c-4330-bfa5-50ca602a28fb" Jul 10 02:08:08.244184 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:08.244147 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:08:08.321218 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:08.321135 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:08:09.006075 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:09.006033 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ll898" Jul 10 02:08:09.006264 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:09.006162 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ll898" podUID="2fea8615-df10-4ccb-84cc-8d382f7bdafe" Jul 10 02:08:09.006264 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:09.006191 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hfgs" Jul 10 02:08:09.006359 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:09.006318 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6hfgs" podUID="8bf086af-02b9-43f2-935f-474133fef4cf" Jul 10 02:08:10.005537 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:10.005506 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcfzz" Jul 10 02:08:10.006034 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:10.005658 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcfzz" podUID="709370a6-295c-4330-bfa5-50ca602a28fb" Jul 10 02:08:10.083206 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:10.083147 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:08:10.147026 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:10.146993 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:08:11.005498 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:11.005457 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ll898" Jul 10 02:08:11.005839 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:11.005457 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hfgs" Jul 10 02:08:11.005839 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:11.005587 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ll898" podUID="2fea8615-df10-4ccb-84cc-8d382f7bdafe" Jul 10 02:08:11.005839 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:11.005717 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6hfgs" podUID="8bf086af-02b9-43f2-935f-474133fef4cf" Jul 10 02:08:12.005281 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:12.005240 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcfzz" Jul 10 02:08:12.005567 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:12.005398 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcfzz" podUID="709370a6-295c-4330-bfa5-50ca602a28fb" Jul 10 02:08:13.005349 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:13.005312 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hfgs" Jul 10 02:08:13.005844 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:13.005310 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ll898" Jul 10 02:08:13.005844 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:13.005468 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6hfgs" podUID="8bf086af-02b9-43f2-935f-474133fef4cf" Jul 10 02:08:13.005844 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:13.005528 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ll898" podUID="2fea8615-df10-4ccb-84cc-8d382f7bdafe" Jul 10 02:08:14.005464 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:14.005427 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcfzz" Jul 10 02:08:14.005859 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:14.005549 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcfzz" podUID="709370a6-295c-4330-bfa5-50ca602a28fb" Jul 10 02:08:14.211712 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:14.211455 2566 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 10 02:08:14.879870 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:14.879491 2566 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-10T02:08:14.211703304Z","UUID":"80234f4d-1726-4c5e-b1d6-42c567164440","Handler":null,"Name":"","Endpoint":""} Jul 10 02:08:14.881401 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:14.881376 2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 10 02:08:14.881554 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:14.881424 2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 10 02:08:15.008346 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:15.008316 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ll898" Jul 10 02:08:15.008346 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:15.008327 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hfgs" Jul 10 02:08:15.008875 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:15.008408 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ll898" podUID="2fea8615-df10-4ccb-84cc-8d382f7bdafe" Jul 10 02:08:15.008875 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:15.008533 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6hfgs" podUID="8bf086af-02b9-43f2-935f-474133fef4cf" Jul 10 02:08:15.101663 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:15.101633 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vtbg2" event={"ID":"0fe32bca-a055-4658-b899-d22fbd705e41","Type":"ContainerStarted","Data":"6df9151023e0394c216797d5fb6dfa2097a8d9176d721cca936a2e2af036a667"} Jul 10 02:08:15.104314 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:15.104283 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" event={"ID":"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc","Type":"ContainerStarted","Data":"7aaa3995dcdfcba4913912ad457fccd90b9d2cb9e0ce777c058c7e033d571740"} Jul 10 02:08:15.104424 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:15.104353 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" event={"ID":"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc","Type":"ContainerStarted","Data":"582774cb462c04fcba7637d98ec695bb5894c4ee339e7d91e3538ee16fbaa3ae"} Jul 10 02:08:15.104424 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:15.104383 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" event={"ID":"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc","Type":"ContainerStarted","Data":"6d6f2b80d8f5b2387844db2a50ec41562973e5b5bbcf7baad73ee095bbcb4216"} Jul 10 02:08:15.104528 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:15.104418 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" event={"ID":"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc","Type":"ContainerStarted","Data":"954b7a9058ae5a75df14fc87d01a6172eb4e561e2747b285e7502d4672b10b4a"} Jul 10 02:08:15.104528 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:15.104457 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" event={"ID":"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc","Type":"ContainerStarted","Data":"02a0bc959cd96dafbd352cc14f72c13ab16bfa717ca679680e64bbc0c6e8fb85"} Jul 10 02:08:15.104528 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:15.104490 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" event={"ID":"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc","Type":"ContainerStarted","Data":"e66ecac25cfa7d8a726687a43f7c4db390056e587e3d3ee8e0433e562a565d4d"} Jul 10 02:08:15.107873 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:15.107846 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wpxd6" event={"ID":"a49e0328-71c1-43ce-80e5-83d0c0862cf9","Type":"ContainerStarted","Data":"d0e995c23b7de7e2e80144a2453492df6269e4dc7c5b9369c8de1bb6bc2487d5"} Jul 10 02:08:15.109509 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:15.109487 2566 generic.go:356] "Generic (PLEG): container finished" podID="fad9e446-daca-42e4-a618-5defb2494fc3" containerID="83e6606e4018091e2c31606b06bbce43826c3a7aea0c6720ac819626aaad4881" exitCode=0 Jul 10 02:08:15.109636 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:15.109526 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f9csf" event={"ID":"fad9e446-daca-42e4-a618-5defb2494fc3","Type":"ContainerDied","Data":"83e6606e4018091e2c31606b06bbce43826c3a7aea0c6720ac819626aaad4881"} Jul 10 02:08:15.129239 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:15.129187 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-wpxd6" podStartSLOduration=3.184285607 podStartE2EDuration="26.129172132s" totalImagesPullingTime="22.944886525s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 02:07:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:08:15.128691016 +0000 UTC m=+26.704968101" watchObservedRunningTime="2026-07-10 02:08:15.129172132 +0000 UTC m=+26.705449217" Jul 10 02:08:15.259506 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:15.259414 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:08:15.369301 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:15.369268 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:08:15.955983 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:15.955903 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:08:16.004793 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:16.004762 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcfzz" Jul 10 02:08:16.004924 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:16.004862 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcfzz" podUID="709370a6-295c-4330-bfa5-50ca602a28fb" Jul 10 02:08:16.053644 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:16.053592 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:08:16.115001 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:16.114953 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vtbg2" event={"ID":"0fe32bca-a055-4658-b899-d22fbd705e41","Type":"ContainerStarted","Data":"a72c793f751872f89d8298a1469185689ece69c4bc35d580b6db55648c0df271"} Jul 10 02:08:16.134864 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:16.134816 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vtbg2" podStartSLOduration=3.954827902 podStartE2EDuration="27.134793362s" totalImagesPullingTime="23.17996546s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 02:07:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:08:16.133706968 +0000 UTC m=+27.709984051" watchObservedRunningTime="2026-07-10 02:08:16.134793362 +0000 UTC m=+27.711070444" Jul 10 02:08:16.140412 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:16.140379 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:08:17.005346 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:17.005306 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ll898" Jul 10 02:08:17.005544 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:17.005426 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ll898" podUID="2fea8615-df10-4ccb-84cc-8d382f7bdafe" Jul 10 02:08:17.005544 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:17.005480 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hfgs" Jul 10 02:08:17.005656 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:17.005577 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6hfgs" podUID="8bf086af-02b9-43f2-935f-474133fef4cf" Jul 10 02:08:17.120663 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:17.120627 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" event={"ID":"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc","Type":"ContainerStarted","Data":"3f0c8abf52a4c8c5a21c7957dee21ceceb52c02258db68fb1ac9914dc4f4a519"} Jul 10 02:08:17.122809 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:17.122778 2566 generic.go:356] "Generic (PLEG): container finished" podID="fad9e446-daca-42e4-a618-5defb2494fc3" containerID="0f384872f316db1de346befa38fed35c4bfc0fb79cbec2451ffba260e7e193e0" exitCode=0 Jul 10 02:08:17.122952 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:17.122852 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f9csf" event={"ID":"fad9e446-daca-42e4-a618-5defb2494fc3","Type":"ContainerDied","Data":"0f384872f316db1de346befa38fed35c4bfc0fb79cbec2451ffba260e7e193e0"} Jul 10 02:08:17.904614 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:17.904569 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:08:18.004774 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:18.004740 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcfzz" Jul 10 02:08:18.005053 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:18.004865 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcfzz" podUID="709370a6-295c-4330-bfa5-50ca602a28fb" Jul 10 02:08:18.005053 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:18.005036 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:08:19.005775 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:19.005744 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ll898" Jul 10 02:08:19.006241 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:19.005823 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ll898" podUID="2fea8615-df10-4ccb-84cc-8d382f7bdafe" Jul 10 02:08:19.006241 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:19.005901 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hfgs" Jul 10 02:08:19.006241 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:19.005986 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6hfgs" podUID="8bf086af-02b9-43f2-935f-474133fef4cf" Jul 10 02:08:19.129668 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:19.129640 2566 generic.go:356] "Generic (PLEG): container finished" podID="fad9e446-daca-42e4-a618-5defb2494fc3" containerID="ca9dcecd7758c99802c946b7aa06d2c87188012b6e044c69bab1a2ee3709e82d" exitCode=0 Jul 10 02:08:19.129808 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:19.129680 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f9csf" event={"ID":"fad9e446-daca-42e4-a618-5defb2494fc3","Type":"ContainerDied","Data":"ca9dcecd7758c99802c946b7aa06d2c87188012b6e044c69bab1a2ee3709e82d"} Jul 10 02:08:20.005617 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:20.005419 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcfzz" Jul 10 02:08:20.005787 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:20.005704 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcfzz" podUID="709370a6-295c-4330-bfa5-50ca602a28fb" Jul 10 02:08:20.137472 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:20.137423 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" event={"ID":"5cf43a6d-60e3-4c7d-9d72-ecb5a0a68afc","Type":"ContainerStarted","Data":"699b0358591ca18d962e15bec7102102612b8bfb16f8f048141474b400ba2a28"} Jul 10 02:08:20.138079 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:20.137632 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:08:20.138079 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:20.137688 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:08:20.138079 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:20.137739 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:08:20.156693 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:20.156665 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:08:20.156818 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:20.156765 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:08:20.166655 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:20.166611 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" podStartSLOduration=8.209002016 podStartE2EDuration="31.166579604s" totalImagesPullingTime="22.957577588s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 02:07:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:08:20.166260574 +0000 UTC m=+31.742537668" watchObservedRunningTime="2026-07-10 02:08:20.166579604 +0000 UTC m=+31.742856690" Jul 10 02:08:21.005249 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:21.005220 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hfgs" Jul 10 02:08:21.005415 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:21.005340 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6hfgs" podUID="8bf086af-02b9-43f2-935f-474133fef4cf" Jul 10 02:08:21.005415 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:21.005394 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ll898" Jul 10 02:08:21.005533 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:21.005512 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ll898" podUID="2fea8615-df10-4ccb-84cc-8d382f7bdafe" Jul 10 02:08:21.537002 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:21.536690 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pcfzz"] Jul 10 02:08:21.537002 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:21.536806 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6hfgs"] Jul 10 02:08:21.537002 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:21.536847 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ll898"] Jul 10 02:08:21.537002 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:21.536984 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hfgs" Jul 10 02:08:21.537002 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:21.536990 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ll898" Jul 10 02:08:21.537769 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:21.537142 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6hfgs" podUID="8bf086af-02b9-43f2-935f-474133fef4cf" Jul 10 02:08:21.537769 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:21.537210 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ll898" podUID="2fea8615-df10-4ccb-84cc-8d382f7bdafe" Jul 10 02:08:21.537769 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:21.537235 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcfzz" Jul 10 02:08:21.537769 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:21.537368 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcfzz" podUID="709370a6-295c-4330-bfa5-50ca602a28fb" Jul 10 02:08:22.676998 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:22.676956 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bf086af-02b9-43f2-935f-474133fef4cf-metrics-certs\") pod \"network-metrics-daemon-6hfgs\" (UID: \"8bf086af-02b9-43f2-935f-474133fef4cf\") " pod="openshift-multus/network-metrics-daemon-6hfgs" Jul 10 02:08:22.677361 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:22.677008 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/709370a6-295c-4330-bfa5-50ca602a28fb-original-pull-secret\") pod \"global-pull-secret-syncer-pcfzz\" (UID: \"709370a6-295c-4330-bfa5-50ca602a28fb\") " pod="kube-system/global-pull-secret-syncer-pcfzz" Jul 10 02:08:22.677361 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:22.677109 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 02:08:22.677361 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:22.677121 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 02:08:22.677361 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:22.677169 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/709370a6-295c-4330-bfa5-50ca602a28fb-original-pull-secret podName:709370a6-295c-4330-bfa5-50ca602a28fb nodeName:}" failed. No retries permitted until 2026-07-10 02:08:54.677155037 +0000 UTC m=+66.253432104 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/709370a6-295c-4330-bfa5-50ca602a28fb-original-pull-secret") pod "global-pull-secret-syncer-pcfzz" (UID: "709370a6-295c-4330-bfa5-50ca602a28fb") : object "kube-system"/"original-pull-secret" not registered Jul 10 02:08:22.677361 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:22.677187 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bf086af-02b9-43f2-935f-474133fef4cf-metrics-certs podName:8bf086af-02b9-43f2-935f-474133fef4cf nodeName:}" failed. No retries permitted until 2026-07-10 02:08:54.677178477 +0000 UTC m=+66.253455542 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8bf086af-02b9-43f2-935f-474133fef4cf-metrics-certs") pod "network-metrics-daemon-6hfgs" (UID: "8bf086af-02b9-43f2-935f-474133fef4cf") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 02:08:22.777328 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:22.777293 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfmw4\" (UniqueName: \"kubernetes.io/projected/2fea8615-df10-4ccb-84cc-8d382f7bdafe-kube-api-access-lfmw4\") pod \"network-check-target-ll898\" (UID: \"2fea8615-df10-4ccb-84cc-8d382f7bdafe\") " pod="openshift-network-diagnostics/network-check-target-ll898" Jul 10 02:08:22.777509 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:22.777491 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 02:08:22.777573 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:22.777521 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 02:08:22.777573 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:22.777534 2566 projected.go:196] Error preparing data for projected volume kube-api-access-lfmw4 for pod openshift-network-diagnostics/network-check-target-ll898: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 02:08:22.777693 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:22.777621 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fea8615-df10-4ccb-84cc-8d382f7bdafe-kube-api-access-lfmw4 podName:2fea8615-df10-4ccb-84cc-8d382f7bdafe nodeName:}" failed. No retries permitted until 2026-07-10 02:08:54.777585982 +0000 UTC m=+66.353863066 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-lfmw4" (UniqueName: "kubernetes.io/projected/2fea8615-df10-4ccb-84cc-8d382f7bdafe-kube-api-access-lfmw4") pod "network-check-target-ll898" (UID: "2fea8615-df10-4ccb-84cc-8d382f7bdafe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 02:08:23.005008 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:23.004925 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hfgs" Jul 10 02:08:23.005161 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:23.004925 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcfzz" Jul 10 02:08:23.005161 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:23.005067 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6hfgs" podUID="8bf086af-02b9-43f2-935f-474133fef4cf" Jul 10 02:08:23.005161 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:23.004925 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ll898" Jul 10 02:08:23.005320 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:23.005155 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcfzz" podUID="709370a6-295c-4330-bfa5-50ca602a28fb" Jul 10 02:08:23.005320 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:23.005268 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ll898" podUID="2fea8615-df10-4ccb-84cc-8d382f7bdafe" Jul 10 02:08:25.005096 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:25.004924 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ll898" Jul 10 02:08:25.005448 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:25.004946 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hfgs" Jul 10 02:08:25.005448 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:25.005174 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ll898" podUID="2fea8615-df10-4ccb-84cc-8d382f7bdafe" Jul 10 02:08:25.005448 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:25.005280 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6hfgs" podUID="8bf086af-02b9-43f2-935f-474133fef4cf" Jul 10 02:08:25.005448 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:25.005000 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcfzz" Jul 10 02:08:25.005448 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:25.005352 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcfzz" podUID="709370a6-295c-4330-bfa5-50ca602a28fb" Jul 10 02:08:25.152182 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:25.152156 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f9csf" event={"ID":"fad9e446-daca-42e4-a618-5defb2494fc3","Type":"ContainerStarted","Data":"925018d49866ae227a1d020da7d74f8047c5981c3cb574bd6439aa95f0f035b0"} Jul 10 02:08:26.157283 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:26.157248 2566 generic.go:356] "Generic (PLEG): container finished" podID="fad9e446-daca-42e4-a618-5defb2494fc3" containerID="925018d49866ae227a1d020da7d74f8047c5981c3cb574bd6439aa95f0f035b0" exitCode=0 Jul 10 02:08:26.157760 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:26.157317 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f9csf" event={"ID":"fad9e446-daca-42e4-a618-5defb2494fc3","Type":"ContainerDied","Data":"925018d49866ae227a1d020da7d74f8047c5981c3cb574bd6439aa95f0f035b0"} Jul 10 02:08:26.782545 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:26.782522 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-65.ec2.internal" event="NodeReady" Jul 10 02:08:26.782760 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:26.782697 2566 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 10 02:08:26.827503 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:26.827473 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pl7t5"] Jul 10 02:08:26.829248 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:26.829230 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pl7t5" Jul 10 02:08:26.830460 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:26.830435 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-r7nm4"] Jul 10 02:08:26.831776 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:26.831758 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 10 02:08:26.831953 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:26.831936 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-r7nm4" Jul 10 02:08:26.832017 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:26.831951 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 10 02:08:26.832105 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:26.832076 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-g8kf5\"" Jul 10 02:08:26.834306 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:26.834290 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 10 02:08:26.834904 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:26.834585 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-zpbsc\"" Jul 10 02:08:26.834904 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:26.834653 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 10 02:08:26.834904 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:26.834660 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 10 02:08:26.841625 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:26.841589 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pl7t5"] Jul 10 02:08:26.844374 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:26.844353 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-r7nm4"] Jul 10 02:08:26.929150 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:26.929121 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:08:27.005175 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:27.005142 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcfzz" Jul 10 02:08:27.005311 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:27.005142 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ll898" Jul 10 02:08:27.005422 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:27.005143 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hfgs" Jul 10 02:08:27.006957 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:27.006931 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9ce933c0-a5ae-4f64-83af-ec2eef8371b0-config-volume\") pod \"dns-default-pl7t5\" (UID: \"9ce933c0-a5ae-4f64-83af-ec2eef8371b0\") " pod="openshift-dns/dns-default-pl7t5" Jul 10 02:08:27.007089 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:27.006975 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ce933c0-a5ae-4f64-83af-ec2eef8371b0-tmp-dir\") pod \"dns-default-pl7t5\" (UID: \"9ce933c0-a5ae-4f64-83af-ec2eef8371b0\") " pod="openshift-dns/dns-default-pl7t5" Jul 10 02:08:27.007089 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:27.007006 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ef36e20-1fdf-4327-bf59-80abf4dfb7bd-cert\") pod \"ingress-canary-r7nm4\" (UID: \"2ef36e20-1fdf-4327-bf59-80abf4dfb7bd\") " pod="openshift-ingress-canary/ingress-canary-r7nm4" Jul 10 02:08:27.007089 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:27.007037 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqsst\" (UniqueName: \"kubernetes.io/projected/2ef36e20-1fdf-4327-bf59-80abf4dfb7bd-kube-api-access-rqsst\") pod \"ingress-canary-r7nm4\" (UID: \"2ef36e20-1fdf-4327-bf59-80abf4dfb7bd\") " pod="openshift-ingress-canary/ingress-canary-r7nm4" Jul 10 02:08:27.007235 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:27.007126 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twfvf\" (UniqueName: \"kubernetes.io/projected/9ce933c0-a5ae-4f64-83af-ec2eef8371b0-kube-api-access-twfvf\") pod \"dns-default-pl7t5\" (UID: \"9ce933c0-a5ae-4f64-83af-ec2eef8371b0\") " pod="openshift-dns/dns-default-pl7t5" Jul 10 02:08:27.007235 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:27.007160 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ce933c0-a5ae-4f64-83af-ec2eef8371b0-metrics-tls\") pod \"dns-default-pl7t5\" (UID: \"9ce933c0-a5ae-4f64-83af-ec2eef8371b0\") " pod="openshift-dns/dns-default-pl7t5" Jul 10 02:08:27.008771 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:27.008748 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 02:08:27.008917 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:27.008897 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 02:08:27.008990 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:27.008929 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-chbtx\"" Jul 10 02:08:27.008990 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:27.008905 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 02:08:27.008990 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:27.008952 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 02:08:27.009135 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:27.009088 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dzm46\"" Jul 10 02:08:27.030273 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:27.030242 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:08:27.108244 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:27.108168 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9ce933c0-a5ae-4f64-83af-ec2eef8371b0-config-volume\") pod \"dns-default-pl7t5\" (UID: \"9ce933c0-a5ae-4f64-83af-ec2eef8371b0\") " pod="openshift-dns/dns-default-pl7t5" Jul 10 02:08:27.108244 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:27.108204 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ce933c0-a5ae-4f64-83af-ec2eef8371b0-tmp-dir\") pod \"dns-default-pl7t5\" (UID: \"9ce933c0-a5ae-4f64-83af-ec2eef8371b0\") " pod="openshift-dns/dns-default-pl7t5" Jul 10 02:08:27.108244 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:27.108226 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ef36e20-1fdf-4327-bf59-80abf4dfb7bd-cert\") pod \"ingress-canary-r7nm4\" (UID: \"2ef36e20-1fdf-4327-bf59-80abf4dfb7bd\") " pod="openshift-ingress-canary/ingress-canary-r7nm4" Jul 10 02:08:27.108244 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:27.108241 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqsst\" (UniqueName: \"kubernetes.io/projected/2ef36e20-1fdf-4327-bf59-80abf4dfb7bd-kube-api-access-rqsst\") pod \"ingress-canary-r7nm4\" (UID: \"2ef36e20-1fdf-4327-bf59-80abf4dfb7bd\") " pod="openshift-ingress-canary/ingress-canary-r7nm4" Jul 10 02:08:27.108522 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:27.108424 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twfvf\" (UniqueName: \"kubernetes.io/projected/9ce933c0-a5ae-4f64-83af-ec2eef8371b0-kube-api-access-twfvf\") pod \"dns-default-pl7t5\" (UID: \"9ce933c0-a5ae-4f64-83af-ec2eef8371b0\") " pod="openshift-dns/dns-default-pl7t5" Jul 10 02:08:27.108522 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:27.108473 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ce933c0-a5ae-4f64-83af-ec2eef8371b0-metrics-tls\") pod \"dns-default-pl7t5\" (UID: \"9ce933c0-a5ae-4f64-83af-ec2eef8371b0\") " pod="openshift-dns/dns-default-pl7t5" Jul 10 02:08:27.108643 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:27.108544 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ce933c0-a5ae-4f64-83af-ec2eef8371b0-tmp-dir\") pod \"dns-default-pl7t5\" (UID: \"9ce933c0-a5ae-4f64-83af-ec2eef8371b0\") " pod="openshift-dns/dns-default-pl7t5" Jul 10 02:08:27.108643 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:27.108623 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 02:08:27.108730 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:27.108702 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ce933c0-a5ae-4f64-83af-ec2eef8371b0-metrics-tls podName:9ce933c0-a5ae-4f64-83af-ec2eef8371b0 nodeName:}" failed. No retries permitted until 2026-07-10 02:08:27.608683243 +0000 UTC m=+39.184960324 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ce933c0-a5ae-4f64-83af-ec2eef8371b0-metrics-tls") pod "dns-default-pl7t5" (UID: "9ce933c0-a5ae-4f64-83af-ec2eef8371b0") : secret "dns-default-metrics-tls" not found Jul 10 02:08:27.108867 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:27.108848 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9ce933c0-a5ae-4f64-83af-ec2eef8371b0-config-volume\") pod \"dns-default-pl7t5\" (UID: \"9ce933c0-a5ae-4f64-83af-ec2eef8371b0\") " pod="openshift-dns/dns-default-pl7t5" Jul 10 02:08:27.112557 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:27.112538 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ef36e20-1fdf-4327-bf59-80abf4dfb7bd-cert\") pod \"ingress-canary-r7nm4\" (UID: \"2ef36e20-1fdf-4327-bf59-80abf4dfb7bd\") " pod="openshift-ingress-canary/ingress-canary-r7nm4" Jul 10 02:08:27.117956 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:27.117933 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twfvf\" (UniqueName: \"kubernetes.io/projected/9ce933c0-a5ae-4f64-83af-ec2eef8371b0-kube-api-access-twfvf\") pod \"dns-default-pl7t5\" (UID: \"9ce933c0-a5ae-4f64-83af-ec2eef8371b0\") " pod="openshift-dns/dns-default-pl7t5" Jul 10 02:08:27.118071 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:27.117996 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqsst\" (UniqueName: \"kubernetes.io/projected/2ef36e20-1fdf-4327-bf59-80abf4dfb7bd-kube-api-access-rqsst\") pod \"ingress-canary-r7nm4\" (UID: \"2ef36e20-1fdf-4327-bf59-80abf4dfb7bd\") " pod="openshift-ingress-canary/ingress-canary-r7nm4" Jul 10 02:08:27.145991 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:27.145960 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-r7nm4" Jul 10 02:08:27.163450 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:27.163421 2566 generic.go:356] "Generic (PLEG): container finished" podID="fad9e446-daca-42e4-a618-5defb2494fc3" containerID="fb69f76e8324a0d27dec6f673c2c1227cf8a54a820eb96a1e15006c4c58cebc3" exitCode=0 Jul 10 02:08:27.163863 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:27.163475 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f9csf" event={"ID":"fad9e446-daca-42e4-a618-5defb2494fc3","Type":"ContainerDied","Data":"fb69f76e8324a0d27dec6f673c2c1227cf8a54a820eb96a1e15006c4c58cebc3"} Jul 10 02:08:27.293564 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:27.293383 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-r7nm4"] Jul 10 02:08:27.297128 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:08:27.297098 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ef36e20_1fdf_4327_bf59_80abf4dfb7bd.slice/crio-6ed3daac1b522c6724262142d07a0ecf0dd0ef67f046ef3a652d8c3f0dbfb485 WatchSource:0}: Error finding container 6ed3daac1b522c6724262142d07a0ecf0dd0ef67f046ef3a652d8c3f0dbfb485: Status 404 returned error can't find the container with id 6ed3daac1b522c6724262142d07a0ecf0dd0ef67f046ef3a652d8c3f0dbfb485 Jul 10 02:08:27.613410 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:27.613374 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ce933c0-a5ae-4f64-83af-ec2eef8371b0-metrics-tls\") pod \"dns-default-pl7t5\" (UID: \"9ce933c0-a5ae-4f64-83af-ec2eef8371b0\") " pod="openshift-dns/dns-default-pl7t5" Jul 10 02:08:27.613584 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:27.613500 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 02:08:27.613584 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:27.613555 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ce933c0-a5ae-4f64-83af-ec2eef8371b0-metrics-tls podName:9ce933c0-a5ae-4f64-83af-ec2eef8371b0 nodeName:}" failed. No retries permitted until 2026-07-10 02:08:28.613540472 +0000 UTC m=+40.189817536 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ce933c0-a5ae-4f64-83af-ec2eef8371b0-metrics-tls") pod "dns-default-pl7t5" (UID: "9ce933c0-a5ae-4f64-83af-ec2eef8371b0") : secret "dns-default-metrics-tls" not found Jul 10 02:08:28.166433 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:28.166403 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-r7nm4" event={"ID":"2ef36e20-1fdf-4327-bf59-80abf4dfb7bd","Type":"ContainerStarted","Data":"6ed3daac1b522c6724262142d07a0ecf0dd0ef67f046ef3a652d8c3f0dbfb485"} Jul 10 02:08:28.169384 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:28.169357 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f9csf" event={"ID":"fad9e446-daca-42e4-a618-5defb2494fc3","Type":"ContainerStarted","Data":"3fb71d209283beb99f6567bf240348d290c0b8c44314519a61360134c62c2b9e"} Jul 10 02:08:28.194801 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:28.194744 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-f9csf" podStartSLOduration=9.515769492 podStartE2EDuration="39.194722219s" totalImagesPullingTime="29.678952727s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 02:07:49 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:08:28.192731178 +0000 UTC m=+39.769008265" watchObservedRunningTime="2026-07-10 02:08:28.194722219 +0000 UTC m=+39.770999313" Jul 10 02:08:28.320442 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:28.320402 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:08:28.381633 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:28.381577 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:08:28.607450 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:28.607414 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:08:28.621358 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:28.621335 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ce933c0-a5ae-4f64-83af-ec2eef8371b0-metrics-tls\") pod \"dns-default-pl7t5\" (UID: \"9ce933c0-a5ae-4f64-83af-ec2eef8371b0\") " pod="openshift-dns/dns-default-pl7t5" Jul 10 02:08:28.621492 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:28.621478 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 02:08:28.621569 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:28.621558 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ce933c0-a5ae-4f64-83af-ec2eef8371b0-metrics-tls podName:9ce933c0-a5ae-4f64-83af-ec2eef8371b0 nodeName:}" failed. No retries permitted until 2026-07-10 02:08:30.621535702 +0000 UTC m=+42.197812779 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ce933c0-a5ae-4f64-83af-ec2eef8371b0-metrics-tls") pod "dns-default-pl7t5" (UID: "9ce933c0-a5ae-4f64-83af-ec2eef8371b0") : secret "dns-default-metrics-tls" not found Jul 10 02:08:29.581358 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:29.581322 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:08:29.640053 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:29.640010 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:08:30.175109 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:30.175027 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-r7nm4" event={"ID":"2ef36e20-1fdf-4327-bf59-80abf4dfb7bd","Type":"ContainerStarted","Data":"63e730efcddf97ee6e479b1a8366368bd20b243d5e18301ce07dd3155b570f1a"} Jul 10 02:08:30.192797 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:30.192746 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-r7nm4" podStartSLOduration=1.7247711 podStartE2EDuration="4.192730064s" totalImagesPullingTime="2.467958964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:08:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:08:30.192014843 +0000 UTC m=+41.768291942" watchObservedRunningTime="2026-07-10 02:08:30.192730064 +0000 UTC m=+41.769007149" Jul 10 02:08:30.635265 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:30.635225 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ce933c0-a5ae-4f64-83af-ec2eef8371b0-metrics-tls\") pod \"dns-default-pl7t5\" (UID: \"9ce933c0-a5ae-4f64-83af-ec2eef8371b0\") " pod="openshift-dns/dns-default-pl7t5" Jul 10 02:08:30.635675 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:30.635350 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 02:08:30.635675 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:30.635410 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ce933c0-a5ae-4f64-83af-ec2eef8371b0-metrics-tls podName:9ce933c0-a5ae-4f64-83af-ec2eef8371b0 nodeName:}" failed. No retries permitted until 2026-07-10 02:08:34.635396802 +0000 UTC m=+46.211673867 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ce933c0-a5ae-4f64-83af-ec2eef8371b0-metrics-tls") pod "dns-default-pl7t5" (UID: "9ce933c0-a5ae-4f64-83af-ec2eef8371b0") : secret "dns-default-metrics-tls" not found Jul 10 02:08:33.981287 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:33.981103 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:08:34.049322 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:34.049280 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:08:34.389390 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:34.389352 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:08:34.491074 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:34.491042 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:08:34.663342 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:34.663262 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ce933c0-a5ae-4f64-83af-ec2eef8371b0-metrics-tls\") pod \"dns-default-pl7t5\" (UID: \"9ce933c0-a5ae-4f64-83af-ec2eef8371b0\") " pod="openshift-dns/dns-default-pl7t5" Jul 10 02:08:34.663486 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:34.663413 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 02:08:34.663530 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:34.663486 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ce933c0-a5ae-4f64-83af-ec2eef8371b0-metrics-tls podName:9ce933c0-a5ae-4f64-83af-ec2eef8371b0 nodeName:}" failed. No retries permitted until 2026-07-10 02:08:42.663468472 +0000 UTC m=+54.239745577 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ce933c0-a5ae-4f64-83af-ec2eef8371b0-metrics-tls") pod "dns-default-pl7t5" (UID: "9ce933c0-a5ae-4f64-83af-ec2eef8371b0") : secret "dns-default-metrics-tls" not found Jul 10 02:08:36.071879 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:36.071845 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:08:42.722190 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:42.722144 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ce933c0-a5ae-4f64-83af-ec2eef8371b0-metrics-tls\") pod \"dns-default-pl7t5\" (UID: \"9ce933c0-a5ae-4f64-83af-ec2eef8371b0\") " pod="openshift-dns/dns-default-pl7t5" Jul 10 02:08:42.722873 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:42.722329 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 02:08:42.722873 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:42.722411 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ce933c0-a5ae-4f64-83af-ec2eef8371b0-metrics-tls podName:9ce933c0-a5ae-4f64-83af-ec2eef8371b0 nodeName:}" failed. No retries permitted until 2026-07-10 02:08:58.722390982 +0000 UTC m=+70.298668065 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ce933c0-a5ae-4f64-83af-ec2eef8371b0-metrics-tls") pod "dns-default-pl7t5" (UID: "9ce933c0-a5ae-4f64-83af-ec2eef8371b0") : secret "dns-default-metrics-tls" not found Jul 10 02:08:52.153112 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:52.153082 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d8d5l" Jul 10 02:08:52.317097 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:52.317064 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:08:52.375429 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:52.375382 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:08:52.953856 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:52.953703 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:08:53.014131 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:53.014096 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:08:54.710956 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:54.710905 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bf086af-02b9-43f2-935f-474133fef4cf-metrics-certs\") pod \"network-metrics-daemon-6hfgs\" (UID: \"8bf086af-02b9-43f2-935f-474133fef4cf\") " pod="openshift-multus/network-metrics-daemon-6hfgs" Jul 10 02:08:54.711422 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:54.710972 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/709370a6-295c-4330-bfa5-50ca602a28fb-original-pull-secret\") pod \"global-pull-secret-syncer-pcfzz\" (UID: \"709370a6-295c-4330-bfa5-50ca602a28fb\") " pod="kube-system/global-pull-secret-syncer-pcfzz" Jul 10 02:08:54.714254 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:54.714232 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 02:08:54.714338 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:54.714240 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 02:08:54.721418 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:54.721399 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 10 02:08:54.721463 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:54.721457 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bf086af-02b9-43f2-935f-474133fef4cf-metrics-certs podName:8bf086af-02b9-43f2-935f-474133fef4cf nodeName:}" failed. No retries permitted until 2026-07-10 02:09:58.721442522 +0000 UTC m=+130.297719599 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8bf086af-02b9-43f2-935f-474133fef4cf-metrics-certs") pod "network-metrics-daemon-6hfgs" (UID: "8bf086af-02b9-43f2-935f-474133fef4cf") : secret "metrics-daemon-secret" not found Jul 10 02:08:54.725091 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:54.725057 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/709370a6-295c-4330-bfa5-50ca602a28fb-original-pull-secret\") pod \"global-pull-secret-syncer-pcfzz\" (UID: \"709370a6-295c-4330-bfa5-50ca602a28fb\") " pod="kube-system/global-pull-secret-syncer-pcfzz" Jul 10 02:08:54.811442 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:54.811394 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfmw4\" (UniqueName: \"kubernetes.io/projected/2fea8615-df10-4ccb-84cc-8d382f7bdafe-kube-api-access-lfmw4\") pod \"network-check-target-ll898\" (UID: \"2fea8615-df10-4ccb-84cc-8d382f7bdafe\") " pod="openshift-network-diagnostics/network-check-target-ll898" Jul 10 02:08:54.814712 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:54.814694 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 02:08:54.824625 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:54.824588 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 02:08:54.834892 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:54.834866 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfmw4\" (UniqueName: \"kubernetes.io/projected/2fea8615-df10-4ccb-84cc-8d382f7bdafe-kube-api-access-lfmw4\") pod \"network-check-target-ll898\" (UID: \"2fea8615-df10-4ccb-84cc-8d382f7bdafe\") " pod="openshift-network-diagnostics/network-check-target-ll898" Jul 10 02:08:54.918083 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:54.918027 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcfzz" Jul 10 02:08:54.933351 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:54.933323 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-chbtx\"" Jul 10 02:08:54.940897 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:54.940867 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ll898" Jul 10 02:08:55.072117 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:55.072086 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pcfzz"] Jul 10 02:08:55.076906 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:08:55.076879 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod709370a6_295c_4330_bfa5_50ca602a28fb.slice/crio-969871be265ac04cbd515d754913cbe0352418a4f4badc714191b4ff1af1312c WatchSource:0}: Error finding container 969871be265ac04cbd515d754913cbe0352418a4f4badc714191b4ff1af1312c: Status 404 returned error can't find the container with id 969871be265ac04cbd515d754913cbe0352418a4f4badc714191b4ff1af1312c Jul 10 02:08:55.100336 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:55.100307 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ll898"] Jul 10 02:08:55.103527 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:08:55.103500 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fea8615_df10_4ccb_84cc_8d382f7bdafe.slice/crio-5d95fe373e1d5dcdd2213ba0a63d7b16a967c276c1e88ee295841b6b57e18b3e WatchSource:0}: Error finding container 5d95fe373e1d5dcdd2213ba0a63d7b16a967c276c1e88ee295841b6b57e18b3e: Status 404 returned error can't find the container with id 5d95fe373e1d5dcdd2213ba0a63d7b16a967c276c1e88ee295841b6b57e18b3e Jul 10 02:08:55.230774 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:55.230671 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ll898" event={"ID":"2fea8615-df10-4ccb-84cc-8d382f7bdafe","Type":"ContainerStarted","Data":"5d95fe373e1d5dcdd2213ba0a63d7b16a967c276c1e88ee295841b6b57e18b3e"} Jul 10 02:08:55.231757 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:55.231725 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pcfzz" event={"ID":"709370a6-295c-4330-bfa5-50ca602a28fb","Type":"ContainerStarted","Data":"969871be265ac04cbd515d754913cbe0352418a4f4badc714191b4ff1af1312c"} Jul 10 02:08:58.738128 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:08:58.738086 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ce933c0-a5ae-4f64-83af-ec2eef8371b0-metrics-tls\") pod \"dns-default-pl7t5\" (UID: \"9ce933c0-a5ae-4f64-83af-ec2eef8371b0\") " pod="openshift-dns/dns-default-pl7t5" Jul 10 02:08:58.738573 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:58.738258 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 02:08:58.738573 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:08:58.738336 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ce933c0-a5ae-4f64-83af-ec2eef8371b0-metrics-tls podName:9ce933c0-a5ae-4f64-83af-ec2eef8371b0 nodeName:}" failed. No retries permitted until 2026-07-10 02:09:30.738318639 +0000 UTC m=+102.314595727 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ce933c0-a5ae-4f64-83af-ec2eef8371b0-metrics-tls") pod "dns-default-pl7t5" (UID: "9ce933c0-a5ae-4f64-83af-ec2eef8371b0") : secret "dns-default-metrics-tls" not found Jul 10 02:09:01.249024 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:09:01.248984 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ll898" event={"ID":"2fea8615-df10-4ccb-84cc-8d382f7bdafe","Type":"ContainerStarted","Data":"1b6d5299de15ae33cbbf9a2dbb4641f6cff136d920b48e26a683bed80bbaa6e0"} Jul 10 02:09:01.249398 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:09:01.249060 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ll898" Jul 10 02:09:01.250388 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:09:01.250361 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pcfzz" event={"ID":"709370a6-295c-4330-bfa5-50ca602a28fb","Type":"ContainerStarted","Data":"0cbbda9d2193c15a4881158d9cfa7f13c9dec4035d6eb1c6ffe1cf404f864b9c"} Jul 10 02:09:01.266698 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:09:01.266656 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ll898" podStartSLOduration=67.131151663 podStartE2EDuration="1m12.266643948s" totalImagesPullingTime="5.135492285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:07:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:09:01.266199032 +0000 UTC m=+72.842476133" watchObservedRunningTime="2026-07-10 02:09:01.266643948 +0000 UTC m=+72.842921034" Jul 10 02:09:01.285113 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:09:01.285073 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-pcfzz" podStartSLOduration=67.118197877 podStartE2EDuration="1m12.285062202s" totalImagesPullingTime="5.166864325s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 02:07:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:09:01.284908892 +0000 UTC m=+72.861185979" watchObservedRunningTime="2026-07-10 02:09:01.285062202 +0000 UTC m=+72.861339288" Jul 10 02:09:27.296634 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:09:27.296587 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:09:27.355391 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:09:27.355356 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:09:30.753485 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:09:30.753439 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ce933c0-a5ae-4f64-83af-ec2eef8371b0-metrics-tls\") pod \"dns-default-pl7t5\" (UID: \"9ce933c0-a5ae-4f64-83af-ec2eef8371b0\") " pod="openshift-dns/dns-default-pl7t5" Jul 10 02:09:30.753889 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:09:30.753569 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 02:09:30.753889 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:09:30.753644 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ce933c0-a5ae-4f64-83af-ec2eef8371b0-metrics-tls podName:9ce933c0-a5ae-4f64-83af-ec2eef8371b0 nodeName:}" failed. No retries permitted until 2026-07-10 02:10:34.753629258 +0000 UTC m=+166.329906322 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ce933c0-a5ae-4f64-83af-ec2eef8371b0-metrics-tls") pod "dns-default-pl7t5" (UID: "9ce933c0-a5ae-4f64-83af-ec2eef8371b0") : secret "dns-default-metrics-tls" not found Jul 10 02:09:32.255494 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:09:32.255463 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ll898" Jul 10 02:09:52.646227 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:09:52.646196 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:09:58.753502 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:09:58.753454 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bf086af-02b9-43f2-935f-474133fef4cf-metrics-certs\") pod \"network-metrics-daemon-6hfgs\" (UID: \"8bf086af-02b9-43f2-935f-474133fef4cf\") " pod="openshift-multus/network-metrics-daemon-6hfgs" Jul 10 02:09:58.753938 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:09:58.753648 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 10 02:09:58.753938 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:09:58.753735 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bf086af-02b9-43f2-935f-474133fef4cf-metrics-certs podName:8bf086af-02b9-43f2-935f-474133fef4cf nodeName:}" failed. No retries permitted until 2026-07-10 02:12:00.753717403 +0000 UTC m=+252.329994483 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8bf086af-02b9-43f2-935f-474133fef4cf-metrics-certs") pod "network-metrics-daemon-6hfgs" (UID: "8bf086af-02b9-43f2-935f-474133fef4cf") : secret "metrics-daemon-secret" not found Jul 10 02:10:29.839353 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:10:29.839308 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-pl7t5" podUID="9ce933c0-a5ae-4f64-83af-ec2eef8371b0" Jul 10 02:10:30.024529 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:10:30.024487 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-6hfgs" podUID="8bf086af-02b9-43f2-935f-474133fef4cf" Jul 10 02:10:30.465111 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:10:30.465080 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pl7t5" Jul 10 02:10:34.804004 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:10:34.803966 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ce933c0-a5ae-4f64-83af-ec2eef8371b0-metrics-tls\") pod \"dns-default-pl7t5\" (UID: \"9ce933c0-a5ae-4f64-83af-ec2eef8371b0\") " pod="openshift-dns/dns-default-pl7t5" Jul 10 02:10:34.804391 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:10:34.804082 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 02:10:34.804391 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:10:34.804135 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ce933c0-a5ae-4f64-83af-ec2eef8371b0-metrics-tls podName:9ce933c0-a5ae-4f64-83af-ec2eef8371b0 nodeName:}" failed. No retries permitted until 2026-07-10 02:12:36.804122197 +0000 UTC m=+288.380399260 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ce933c0-a5ae-4f64-83af-ec2eef8371b0-metrics-tls") pod "dns-default-pl7t5" (UID: "9ce933c0-a5ae-4f64-83af-ec2eef8371b0") : secret "dns-default-metrics-tls" not found Jul 10 02:10:43.005387 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:10:43.005297 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hfgs" Jul 10 02:10:45.121288 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:10:45.121249 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:11:42.009325 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:42.009280 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:11:45.108387 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.108361 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-fn4xn"] Jul 10 02:11:45.110194 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.110178 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-fn4xn" Jul 10 02:11:45.112705 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.112683 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5z9lh"] Jul 10 02:11:45.114439 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.114415 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5z9lh" Jul 10 02:11:45.122033 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.122018 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 10 02:11:45.123614 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.123573 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 10 02:11:45.123995 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.123979 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 10 02:11:45.125007 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.124991 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-sf8x7\"" Jul 10 02:11:45.125007 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.125001 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 10 02:11:45.125124 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.125009 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 10 02:11:45.126694 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.126675 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 10 02:11:45.127198 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.127181 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-fn4xn"] Jul 10 02:11:45.130067 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.130049 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-f2lvp\"" Jul 10 02:11:45.138266 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.138247 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5z9lh"] Jul 10 02:11:45.201792 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.201771 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-8ghxz"] Jul 10 02:11:45.203473 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.203457 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-8ghxz" Jul 10 02:11:45.205340 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.205315 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-whpts"] Jul 10 02:11:45.206532 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.206509 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-pfdvp\"" Jul 10 02:11:45.206633 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.206554 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 10 02:11:45.206633 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.206586 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 10 02:11:45.206731 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.206629 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 10 02:11:45.207157 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.207140 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6cd68c6448-czfxh"] Jul 10 02:11:45.207247 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.207234 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-whpts" Jul 10 02:11:45.208708 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.208691 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6cd68c6448-czfxh" Jul 10 02:11:45.210814 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.210760 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 10 02:11:45.210814 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.210777 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 10 02:11:45.210814 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.210791 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 10 02:11:45.211380 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.211368 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 10 02:11:45.212016 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.212003 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-7gnkk\"" Jul 10 02:11:45.212466 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.212450 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 10 02:11:45.212521 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.212474 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 10 02:11:45.212568 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.212552 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 10 02:11:45.212568 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.212562 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 10 02:11:45.212663 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.212570 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-frwtg\"" Jul 10 02:11:45.212781 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.212766 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 10 02:11:45.212878 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.212862 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 10 02:11:45.216829 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.216808 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-8ghxz"] Jul 10 02:11:45.223455 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.223327 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-whpts"] Jul 10 02:11:45.225365 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.225344 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6cd68c6448-czfxh"] Jul 10 02:11:45.257362 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.257340 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ebeadbc-c34a-4aae-aeab-27ea702d6cf3-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-8ghxz\" (UID: \"2ebeadbc-c34a-4aae-aeab-27ea702d6cf3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-8ghxz" Jul 10 02:11:45.257464 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.257363 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zd6d\" (UniqueName: \"kubernetes.io/projected/2ebeadbc-c34a-4aae-aeab-27ea702d6cf3-kube-api-access-7zd6d\") pod \"cluster-samples-operator-65db858894-8ghxz\" (UID: \"2ebeadbc-c34a-4aae-aeab-27ea702d6cf3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-8ghxz" Jul 10 02:11:45.257464 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.257380 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/82ce150d-af95-4650-a996-fbe49696196f-stats-auth\") pod \"router-default-6cd68c6448-czfxh\" (UID: \"82ce150d-af95-4650-a996-fbe49696196f\") " pod="openshift-ingress/router-default-6cd68c6448-czfxh" Jul 10 02:11:45.257464 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.257401 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82ce150d-af95-4650-a996-fbe49696196f-metrics-certs\") pod \"router-default-6cd68c6448-czfxh\" (UID: \"82ce150d-af95-4650-a996-fbe49696196f\") " pod="openshift-ingress/router-default-6cd68c6448-czfxh" Jul 10 02:11:45.257464 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.257439 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/397d53aa-cc3d-4319-8d59-10127a7c6883-serving-cert\") pod \"service-ca-operator-7695fc6c8c-5z9lh\" (UID: \"397d53aa-cc3d-4319-8d59-10127a7c6883\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5z9lh" Jul 10 02:11:45.257658 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.257474 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82ce150d-af95-4650-a996-fbe49696196f-service-ca-bundle\") pod \"router-default-6cd68c6448-czfxh\" (UID: \"82ce150d-af95-4650-a996-fbe49696196f\") " pod="openshift-ingress/router-default-6cd68c6448-czfxh" Jul 10 02:11:45.257658 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.257497 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/397d53aa-cc3d-4319-8d59-10127a7c6883-config\") pod \"service-ca-operator-7695fc6c8c-5z9lh\" (UID: \"397d53aa-cc3d-4319-8d59-10127a7c6883\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5z9lh" Jul 10 02:11:45.257658 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.257534 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhsmt\" (UniqueName: \"kubernetes.io/projected/397d53aa-cc3d-4319-8d59-10127a7c6883-kube-api-access-qhsmt\") pod \"service-ca-operator-7695fc6c8c-5z9lh\" (UID: \"397d53aa-cc3d-4319-8d59-10127a7c6883\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5z9lh" Jul 10 02:11:45.257658 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.257550 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/54ca438c-c22e-45df-8aa2-f653453022b0-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-whpts\" (UID: \"54ca438c-c22e-45df-8aa2-f653453022b0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-whpts" Jul 10 02:11:45.257658 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.257570 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfc9r\" (UniqueName: \"kubernetes.io/projected/54ca438c-c22e-45df-8aa2-f653453022b0-kube-api-access-jfc9r\") pod \"kube-storage-version-migrator-operator-799cb7f958-whpts\" (UID: \"54ca438c-c22e-45df-8aa2-f653453022b0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-whpts" Jul 10 02:11:45.257658 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.257609 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/82ce150d-af95-4650-a996-fbe49696196f-default-certificate\") pod \"router-default-6cd68c6448-czfxh\" (UID: \"82ce150d-af95-4650-a996-fbe49696196f\") " pod="openshift-ingress/router-default-6cd68c6448-czfxh" Jul 10 02:11:45.257658 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.257633 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmsdj\" (UniqueName: \"kubernetes.io/projected/82ce150d-af95-4650-a996-fbe49696196f-kube-api-access-bmsdj\") pod \"router-default-6cd68c6448-czfxh\" (UID: \"82ce150d-af95-4650-a996-fbe49696196f\") " pod="openshift-ingress/router-default-6cd68c6448-czfxh" Jul 10 02:11:45.257658 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.257652 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt7t5\" (UniqueName: \"kubernetes.io/projected/1cddfa1b-4e04-4165-99ff-24071eb16422-kube-api-access-pt7t5\") pod \"volume-data-source-validator-78495597f6-fn4xn\" (UID: \"1cddfa1b-4e04-4165-99ff-24071eb16422\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-fn4xn" Jul 10 02:11:45.257934 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.257681 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/54ca438c-c22e-45df-8aa2-f653453022b0-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-whpts\" (UID: \"54ca438c-c22e-45df-8aa2-f653453022b0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-whpts" Jul 10 02:11:45.358920 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.358866 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ebeadbc-c34a-4aae-aeab-27ea702d6cf3-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-8ghxz\" (UID: \"2ebeadbc-c34a-4aae-aeab-27ea702d6cf3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-8ghxz" Jul 10 02:11:45.358920 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.358891 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zd6d\" (UniqueName: \"kubernetes.io/projected/2ebeadbc-c34a-4aae-aeab-27ea702d6cf3-kube-api-access-7zd6d\") pod \"cluster-samples-operator-65db858894-8ghxz\" (UID: \"2ebeadbc-c34a-4aae-aeab-27ea702d6cf3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-8ghxz" Jul 10 02:11:45.358920 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.358909 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/82ce150d-af95-4650-a996-fbe49696196f-stats-auth\") pod \"router-default-6cd68c6448-czfxh\" (UID: \"82ce150d-af95-4650-a996-fbe49696196f\") " pod="openshift-ingress/router-default-6cd68c6448-czfxh" Jul 10 02:11:45.359077 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:11:45.359030 2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 02:11:45.359264 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.358924 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82ce150d-af95-4650-a996-fbe49696196f-metrics-certs\") pod \"router-default-6cd68c6448-czfxh\" (UID: \"82ce150d-af95-4650-a996-fbe49696196f\") " pod="openshift-ingress/router-default-6cd68c6448-czfxh" Jul 10 02:11:45.359335 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.359293 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/397d53aa-cc3d-4319-8d59-10127a7c6883-serving-cert\") pod \"service-ca-operator-7695fc6c8c-5z9lh\" (UID: \"397d53aa-cc3d-4319-8d59-10127a7c6883\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5z9lh" Jul 10 02:11:45.359384 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.359334 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82ce150d-af95-4650-a996-fbe49696196f-service-ca-bundle\") pod \"router-default-6cd68c6448-czfxh\" (UID: \"82ce150d-af95-4650-a996-fbe49696196f\") " pod="openshift-ingress/router-default-6cd68c6448-czfxh" Jul 10 02:11:45.359384 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:11:45.359339 2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 02:11:45.359484 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:11:45.359387 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ebeadbc-c34a-4aae-aeab-27ea702d6cf3-samples-operator-tls podName:2ebeadbc-c34a-4aae-aeab-27ea702d6cf3 nodeName:}" failed. No retries permitted until 2026-07-10 02:11:45.859360684 +0000 UTC m=+237.435637798 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2ebeadbc-c34a-4aae-aeab-27ea702d6cf3-samples-operator-tls") pod "cluster-samples-operator-65db858894-8ghxz" (UID: "2ebeadbc-c34a-4aae-aeab-27ea702d6cf3") : secret "samples-operator-tls" not found Jul 10 02:11:45.359484 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.359430 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/397d53aa-cc3d-4319-8d59-10127a7c6883-config\") pod \"service-ca-operator-7695fc6c8c-5z9lh\" (UID: \"397d53aa-cc3d-4319-8d59-10127a7c6883\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5z9lh" Jul 10 02:11:45.359588 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.359511 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhsmt\" (UniqueName: \"kubernetes.io/projected/397d53aa-cc3d-4319-8d59-10127a7c6883-kube-api-access-qhsmt\") pod \"service-ca-operator-7695fc6c8c-5z9lh\" (UID: \"397d53aa-cc3d-4319-8d59-10127a7c6883\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5z9lh" Jul 10 02:11:45.359588 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.359550 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/54ca438c-c22e-45df-8aa2-f653453022b0-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-whpts\" (UID: \"54ca438c-c22e-45df-8aa2-f653453022b0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-whpts" Jul 10 02:11:45.359744 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.359725 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfc9r\" (UniqueName: \"kubernetes.io/projected/54ca438c-c22e-45df-8aa2-f653453022b0-kube-api-access-jfc9r\") pod \"kube-storage-version-migrator-operator-799cb7f958-whpts\" (UID: \"54ca438c-c22e-45df-8aa2-f653453022b0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-whpts" Jul 10 02:11:45.359819 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:11:45.359802 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82ce150d-af95-4650-a996-fbe49696196f-service-ca-bundle podName:82ce150d-af95-4650-a996-fbe49696196f nodeName:}" failed. No retries permitted until 2026-07-10 02:11:45.859782147 +0000 UTC m=+237.436059234 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/82ce150d-af95-4650-a996-fbe49696196f-service-ca-bundle") pod "router-default-6cd68c6448-czfxh" (UID: "82ce150d-af95-4650-a996-fbe49696196f") : configmap references non-existent config key: service-ca.crt Jul 10 02:11:45.359917 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:11:45.359897 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82ce150d-af95-4650-a996-fbe49696196f-metrics-certs podName:82ce150d-af95-4650-a996-fbe49696196f nodeName:}" failed. No retries permitted until 2026-07-10 02:11:45.859878806 +0000 UTC m=+237.436155886 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82ce150d-af95-4650-a996-fbe49696196f-metrics-certs") pod "router-default-6cd68c6448-czfxh" (UID: "82ce150d-af95-4650-a996-fbe49696196f") : secret "router-metrics-certs-default" not found Jul 10 02:11:45.360581 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.360385 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/397d53aa-cc3d-4319-8d59-10127a7c6883-config\") pod \"service-ca-operator-7695fc6c8c-5z9lh\" (UID: \"397d53aa-cc3d-4319-8d59-10127a7c6883\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5z9lh" Jul 10 02:11:45.360711 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.360693 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/82ce150d-af95-4650-a996-fbe49696196f-default-certificate\") pod \"router-default-6cd68c6448-czfxh\" (UID: \"82ce150d-af95-4650-a996-fbe49696196f\") " pod="openshift-ingress/router-default-6cd68c6448-czfxh" Jul 10 02:11:45.362557 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.362535 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmsdj\" (UniqueName: \"kubernetes.io/projected/82ce150d-af95-4650-a996-fbe49696196f-kube-api-access-bmsdj\") pod \"router-default-6cd68c6448-czfxh\" (UID: \"82ce150d-af95-4650-a996-fbe49696196f\") " pod="openshift-ingress/router-default-6cd68c6448-czfxh" Jul 10 02:11:45.362648 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.362588 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt7t5\" (UniqueName: \"kubernetes.io/projected/1cddfa1b-4e04-4165-99ff-24071eb16422-kube-api-access-pt7t5\") pod \"volume-data-source-validator-78495597f6-fn4xn\" (UID: \"1cddfa1b-4e04-4165-99ff-24071eb16422\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-fn4xn" Jul 10 02:11:45.362714 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.362664 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/54ca438c-c22e-45df-8aa2-f653453022b0-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-whpts\" (UID: \"54ca438c-c22e-45df-8aa2-f653453022b0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-whpts" Jul 10 02:11:45.362965 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.362940 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/54ca438c-c22e-45df-8aa2-f653453022b0-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-whpts\" (UID: \"54ca438c-c22e-45df-8aa2-f653453022b0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-whpts" Jul 10 02:11:45.363044 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.362320 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/397d53aa-cc3d-4319-8d59-10127a7c6883-serving-cert\") pod \"service-ca-operator-7695fc6c8c-5z9lh\" (UID: \"397d53aa-cc3d-4319-8d59-10127a7c6883\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5z9lh" Jul 10 02:11:45.363044 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.362982 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/82ce150d-af95-4650-a996-fbe49696196f-default-certificate\") pod \"router-default-6cd68c6448-czfxh\" (UID: \"82ce150d-af95-4650-a996-fbe49696196f\") " pod="openshift-ingress/router-default-6cd68c6448-czfxh" Jul 10 02:11:45.363280 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.363255 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/54ca438c-c22e-45df-8aa2-f653453022b0-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-whpts\" (UID: \"54ca438c-c22e-45df-8aa2-f653453022b0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-whpts" Jul 10 02:11:45.363510 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.363493 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/82ce150d-af95-4650-a996-fbe49696196f-stats-auth\") pod \"router-default-6cd68c6448-czfxh\" (UID: \"82ce150d-af95-4650-a996-fbe49696196f\") " pod="openshift-ingress/router-default-6cd68c6448-czfxh" Jul 10 02:11:45.372056 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.372032 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zd6d\" (UniqueName: \"kubernetes.io/projected/2ebeadbc-c34a-4aae-aeab-27ea702d6cf3-kube-api-access-7zd6d\") pod \"cluster-samples-operator-65db858894-8ghxz\" (UID: \"2ebeadbc-c34a-4aae-aeab-27ea702d6cf3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-8ghxz" Jul 10 02:11:45.372653 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.372618 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfc9r\" (UniqueName: \"kubernetes.io/projected/54ca438c-c22e-45df-8aa2-f653453022b0-kube-api-access-jfc9r\") pod \"kube-storage-version-migrator-operator-799cb7f958-whpts\" (UID: \"54ca438c-c22e-45df-8aa2-f653453022b0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-whpts" Jul 10 02:11:45.373801 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.373780 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhsmt\" (UniqueName: \"kubernetes.io/projected/397d53aa-cc3d-4319-8d59-10127a7c6883-kube-api-access-qhsmt\") pod \"service-ca-operator-7695fc6c8c-5z9lh\" (UID: \"397d53aa-cc3d-4319-8d59-10127a7c6883\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5z9lh" Jul 10 02:11:45.374038 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.374021 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmsdj\" (UniqueName: \"kubernetes.io/projected/82ce150d-af95-4650-a996-fbe49696196f-kube-api-access-bmsdj\") pod \"router-default-6cd68c6448-czfxh\" (UID: \"82ce150d-af95-4650-a996-fbe49696196f\") " pod="openshift-ingress/router-default-6cd68c6448-czfxh" Jul 10 02:11:45.374247 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.374226 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt7t5\" (UniqueName: \"kubernetes.io/projected/1cddfa1b-4e04-4165-99ff-24071eb16422-kube-api-access-pt7t5\") pod \"volume-data-source-validator-78495597f6-fn4xn\" (UID: \"1cddfa1b-4e04-4165-99ff-24071eb16422\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-fn4xn" Jul 10 02:11:45.375013 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.374992 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:11:45.419780 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.419760 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-fn4xn" Jul 10 02:11:45.425045 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.425030 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5z9lh" Jul 10 02:11:45.476416 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.476368 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:11:45.520919 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.520885 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-whpts" Jul 10 02:11:45.550557 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.550530 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-fn4xn"] Jul 10 02:11:45.554081 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:11:45.554050 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cddfa1b_4e04_4165_99ff_24071eb16422.slice/crio-c58041ad701a05a40aa31fb090193a1f9de247d0a0b65dcc09523111b95beb3f WatchSource:0}: Error finding container c58041ad701a05a40aa31fb090193a1f9de247d0a0b65dcc09523111b95beb3f: Status 404 returned error can't find the container with id c58041ad701a05a40aa31fb090193a1f9de247d0a0b65dcc09523111b95beb3f Jul 10 02:11:45.566389 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.566365 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5z9lh"] Jul 10 02:11:45.569258 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:11:45.569231 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod397d53aa_cc3d_4319_8d59_10127a7c6883.slice/crio-b8bd29e34f6d34a21a9c39229f4db817d7ba7eaabd8fc38bdf0d243f54521808 WatchSource:0}: Error finding container b8bd29e34f6d34a21a9c39229f4db817d7ba7eaabd8fc38bdf0d243f54521808: Status 404 returned error can't find the container with id b8bd29e34f6d34a21a9c39229f4db817d7ba7eaabd8fc38bdf0d243f54521808 Jul 10 02:11:45.637202 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.637127 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-fn4xn" event={"ID":"1cddfa1b-4e04-4165-99ff-24071eb16422","Type":"ContainerStarted","Data":"c58041ad701a05a40aa31fb090193a1f9de247d0a0b65dcc09523111b95beb3f"} Jul 10 02:11:45.638175 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.638152 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5z9lh" event={"ID":"397d53aa-cc3d-4319-8d59-10127a7c6883","Type":"ContainerStarted","Data":"b8bd29e34f6d34a21a9c39229f4db817d7ba7eaabd8fc38bdf0d243f54521808"} Jul 10 02:11:45.648302 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.648280 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-whpts"] Jul 10 02:11:45.652280 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:11:45.652259 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54ca438c_c22e_45df_8aa2_f653453022b0.slice/crio-238f69141c48c7ead3221e9698f6732d7ccf07c93165348fcd345b0e45f2faae WatchSource:0}: Error finding container 238f69141c48c7ead3221e9698f6732d7ccf07c93165348fcd345b0e45f2faae: Status 404 returned error can't find the container with id 238f69141c48c7ead3221e9698f6732d7ccf07c93165348fcd345b0e45f2faae Jul 10 02:11:45.867073 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.867040 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ebeadbc-c34a-4aae-aeab-27ea702d6cf3-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-8ghxz\" (UID: \"2ebeadbc-c34a-4aae-aeab-27ea702d6cf3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-8ghxz" Jul 10 02:11:45.867073 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.867074 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82ce150d-af95-4650-a996-fbe49696196f-metrics-certs\") pod \"router-default-6cd68c6448-czfxh\" (UID: \"82ce150d-af95-4650-a996-fbe49696196f\") " pod="openshift-ingress/router-default-6cd68c6448-czfxh" Jul 10 02:11:45.867256 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:45.867096 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82ce150d-af95-4650-a996-fbe49696196f-service-ca-bundle\") pod \"router-default-6cd68c6448-czfxh\" (UID: \"82ce150d-af95-4650-a996-fbe49696196f\") " pod="openshift-ingress/router-default-6cd68c6448-czfxh" Jul 10 02:11:45.867256 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:11:45.867185 2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 02:11:45.867256 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:11:45.867219 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82ce150d-af95-4650-a996-fbe49696196f-service-ca-bundle podName:82ce150d-af95-4650-a996-fbe49696196f nodeName:}" failed. No retries permitted until 2026-07-10 02:11:46.867206425 +0000 UTC m=+238.443483489 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/82ce150d-af95-4650-a996-fbe49696196f-service-ca-bundle") pod "router-default-6cd68c6448-czfxh" (UID: "82ce150d-af95-4650-a996-fbe49696196f") : configmap references non-existent config key: service-ca.crt Jul 10 02:11:45.867256 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:11:45.867191 2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 02:11:45.867256 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:11:45.867233 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ebeadbc-c34a-4aae-aeab-27ea702d6cf3-samples-operator-tls podName:2ebeadbc-c34a-4aae-aeab-27ea702d6cf3 nodeName:}" failed. No retries permitted until 2026-07-10 02:11:46.867227024 +0000 UTC m=+238.443504087 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2ebeadbc-c34a-4aae-aeab-27ea702d6cf3-samples-operator-tls") pod "cluster-samples-operator-65db858894-8ghxz" (UID: "2ebeadbc-c34a-4aae-aeab-27ea702d6cf3") : secret "samples-operator-tls" not found Jul 10 02:11:45.867256 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:11:45.867254 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82ce150d-af95-4650-a996-fbe49696196f-metrics-certs podName:82ce150d-af95-4650-a996-fbe49696196f nodeName:}" failed. No retries permitted until 2026-07-10 02:11:46.86724072 +0000 UTC m=+238.443517802 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82ce150d-af95-4650-a996-fbe49696196f-metrics-certs") pod "router-default-6cd68c6448-czfxh" (UID: "82ce150d-af95-4650-a996-fbe49696196f") : secret "router-metrics-certs-default" not found Jul 10 02:11:46.642178 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:46.642101 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-whpts" event={"ID":"54ca438c-c22e-45df-8aa2-f653453022b0","Type":"ContainerStarted","Data":"238f69141c48c7ead3221e9698f6732d7ccf07c93165348fcd345b0e45f2faae"} Jul 10 02:11:46.875867 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:46.875828 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ebeadbc-c34a-4aae-aeab-27ea702d6cf3-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-8ghxz\" (UID: \"2ebeadbc-c34a-4aae-aeab-27ea702d6cf3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-8ghxz" Jul 10 02:11:46.875867 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:46.875879 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82ce150d-af95-4650-a996-fbe49696196f-metrics-certs\") pod \"router-default-6cd68c6448-czfxh\" (UID: \"82ce150d-af95-4650-a996-fbe49696196f\") " pod="openshift-ingress/router-default-6cd68c6448-czfxh" Jul 10 02:11:46.876085 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:46.875915 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82ce150d-af95-4650-a996-fbe49696196f-service-ca-bundle\") pod \"router-default-6cd68c6448-czfxh\" (UID: \"82ce150d-af95-4650-a996-fbe49696196f\") " pod="openshift-ingress/router-default-6cd68c6448-czfxh" Jul 10 02:11:46.876085 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:11:46.876001 2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 02:11:46.876085 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:11:46.876046 2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 02:11:46.876085 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:11:46.876080 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ebeadbc-c34a-4aae-aeab-27ea702d6cf3-samples-operator-tls podName:2ebeadbc-c34a-4aae-aeab-27ea702d6cf3 nodeName:}" failed. No retries permitted until 2026-07-10 02:11:48.876059484 +0000 UTC m=+240.452336551 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2ebeadbc-c34a-4aae-aeab-27ea702d6cf3-samples-operator-tls") pod "cluster-samples-operator-65db858894-8ghxz" (UID: "2ebeadbc-c34a-4aae-aeab-27ea702d6cf3") : secret "samples-operator-tls" not found Jul 10 02:11:46.876279 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:11:46.876105 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82ce150d-af95-4650-a996-fbe49696196f-service-ca-bundle podName:82ce150d-af95-4650-a996-fbe49696196f nodeName:}" failed. No retries permitted until 2026-07-10 02:11:48.876088362 +0000 UTC m=+240.452365426 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/82ce150d-af95-4650-a996-fbe49696196f-service-ca-bundle") pod "router-default-6cd68c6448-czfxh" (UID: "82ce150d-af95-4650-a996-fbe49696196f") : configmap references non-existent config key: service-ca.crt Jul 10 02:11:46.876279 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:11:46.876127 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82ce150d-af95-4650-a996-fbe49696196f-metrics-certs podName:82ce150d-af95-4650-a996-fbe49696196f nodeName:}" failed. No retries permitted until 2026-07-10 02:11:48.876116898 +0000 UTC m=+240.452393962 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82ce150d-af95-4650-a996-fbe49696196f-metrics-certs") pod "router-default-6cd68c6448-czfxh" (UID: "82ce150d-af95-4650-a996-fbe49696196f") : secret "router-metrics-certs-default" not found Jul 10 02:11:47.646123 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:47.646081 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-fn4xn" event={"ID":"1cddfa1b-4e04-4165-99ff-24071eb16422","Type":"ContainerStarted","Data":"c30c3eae0f72f712d784b1a1ea32f9ce3a0433261786a64105abb6fa8d97c9bd"} Jul 10 02:11:47.664724 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:47.664654 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-fn4xn" podStartSLOduration=1.309406912 podStartE2EDuration="2.66463523s" totalImagesPullingTime="1.355228318s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:11:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:11:47.66397418 +0000 UTC m=+239.240251266" watchObservedRunningTime="2026-07-10 02:11:47.66463523 +0000 UTC m=+239.240912317" Jul 10 02:11:47.755325 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:47.755276 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:11:47.812827 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:47.812765 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:11:48.650921 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:48.650823 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5z9lh" event={"ID":"397d53aa-cc3d-4319-8d59-10127a7c6883","Type":"ContainerStarted","Data":"6e45d10e5536049994844a90b9da4166561864f684c97e768f4667986e8f55d2"} Jul 10 02:11:48.652379 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:48.652351 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-whpts" event={"ID":"54ca438c-c22e-45df-8aa2-f653453022b0","Type":"ContainerStarted","Data":"8f05077f93ef57eddd26015b88403255b1e064050a37f3f4924c85d001389de7"} Jul 10 02:11:48.669925 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:48.669878 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5z9lh" podStartSLOduration=0.883784647 podStartE2EDuration="3.669859384s" totalImagesPullingTime="2.786074737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:11:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:11:48.668355051 +0000 UTC m=+240.244632151" watchObservedRunningTime="2026-07-10 02:11:48.669859384 +0000 UTC m=+240.246136471" Jul 10 02:11:48.686692 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:48.686643 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-whpts" podStartSLOduration=0.985247017 podStartE2EDuration="3.686583116s" totalImagesPullingTime="2.701336099s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:11:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:11:48.685471361 +0000 UTC m=+240.261748448" watchObservedRunningTime="2026-07-10 02:11:48.686583116 +0000 UTC m=+240.262860204" Jul 10 02:11:48.892943 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:48.892911 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ebeadbc-c34a-4aae-aeab-27ea702d6cf3-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-8ghxz\" (UID: \"2ebeadbc-c34a-4aae-aeab-27ea702d6cf3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-8ghxz" Jul 10 02:11:48.893085 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:48.892955 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82ce150d-af95-4650-a996-fbe49696196f-metrics-certs\") pod \"router-default-6cd68c6448-czfxh\" (UID: \"82ce150d-af95-4650-a996-fbe49696196f\") " pod="openshift-ingress/router-default-6cd68c6448-czfxh" Jul 10 02:11:48.893085 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:48.892987 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82ce150d-af95-4650-a996-fbe49696196f-service-ca-bundle\") pod \"router-default-6cd68c6448-czfxh\" (UID: \"82ce150d-af95-4650-a996-fbe49696196f\") " pod="openshift-ingress/router-default-6cd68c6448-czfxh" Jul 10 02:11:48.893085 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:11:48.893023 2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 02:11:48.893085 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:11:48.893077 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ebeadbc-c34a-4aae-aeab-27ea702d6cf3-samples-operator-tls podName:2ebeadbc-c34a-4aae-aeab-27ea702d6cf3 nodeName:}" failed. No retries permitted until 2026-07-10 02:11:52.893061513 +0000 UTC m=+244.469338577 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2ebeadbc-c34a-4aae-aeab-27ea702d6cf3-samples-operator-tls") pod "cluster-samples-operator-65db858894-8ghxz" (UID: "2ebeadbc-c34a-4aae-aeab-27ea702d6cf3") : secret "samples-operator-tls" not found Jul 10 02:11:48.893310 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:11:48.893103 2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 02:11:48.893310 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:11:48.893133 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82ce150d-af95-4650-a996-fbe49696196f-service-ca-bundle podName:82ce150d-af95-4650-a996-fbe49696196f nodeName:}" failed. No retries permitted until 2026-07-10 02:11:52.893118404 +0000 UTC m=+244.469395493 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/82ce150d-af95-4650-a996-fbe49696196f-service-ca-bundle") pod "router-default-6cd68c6448-czfxh" (UID: "82ce150d-af95-4650-a996-fbe49696196f") : configmap references non-existent config key: service-ca.crt Jul 10 02:11:48.893310 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:11:48.893169 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82ce150d-af95-4650-a996-fbe49696196f-metrics-certs podName:82ce150d-af95-4650-a996-fbe49696196f nodeName:}" failed. No retries permitted until 2026-07-10 02:11:52.89315772 +0000 UTC m=+244.469434784 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82ce150d-af95-4650-a996-fbe49696196f-metrics-certs") pod "router-default-6cd68c6448-czfxh" (UID: "82ce150d-af95-4650-a996-fbe49696196f") : secret "router-metrics-certs-default" not found Jul 10 02:11:49.344744 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:49.344716 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2w28f"] Jul 10 02:11:49.346832 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:49.346809 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2w28f" Jul 10 02:11:49.349913 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:49.349889 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 10 02:11:49.350048 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:49.349949 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 10 02:11:49.350253 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:49.350239 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pk9p4\"" Jul 10 02:11:49.351407 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:49.351391 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 10 02:11:49.351407 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:49.351404 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 10 02:11:49.361187 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:49.361165 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2w28f"] Jul 10 02:11:49.379602 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:49.379580 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:11:49.448149 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:49.448110 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:11:49.498250 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:49.498226 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2w28f\" (UID: \"7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8\") " pod="openshift-insights/insights-runtime-extractor-2w28f" Jul 10 02:11:49.498354 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:49.498262 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8-data-volume\") pod \"insights-runtime-extractor-2w28f\" (UID: \"7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8\") " pod="openshift-insights/insights-runtime-extractor-2w28f" Jul 10 02:11:49.498354 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:49.498284 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8-crio-socket\") pod \"insights-runtime-extractor-2w28f\" (UID: \"7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8\") " pod="openshift-insights/insights-runtime-extractor-2w28f" Jul 10 02:11:49.498420 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:49.498376 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2w28f\" (UID: \"7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8\") " pod="openshift-insights/insights-runtime-extractor-2w28f" Jul 10 02:11:49.498454 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:49.498432 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jclj2\" (UniqueName: \"kubernetes.io/projected/7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8-kube-api-access-jclj2\") pod \"insights-runtime-extractor-2w28f\" (UID: \"7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8\") " pod="openshift-insights/insights-runtime-extractor-2w28f" Jul 10 02:11:49.599270 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:49.599208 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jclj2\" (UniqueName: \"kubernetes.io/projected/7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8-kube-api-access-jclj2\") pod \"insights-runtime-extractor-2w28f\" (UID: \"7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8\") " pod="openshift-insights/insights-runtime-extractor-2w28f" Jul 10 02:11:49.599375 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:49.599269 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2w28f\" (UID: \"7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8\") " pod="openshift-insights/insights-runtime-extractor-2w28f" Jul 10 02:11:49.599375 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:49.599298 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8-data-volume\") pod \"insights-runtime-extractor-2w28f\" (UID: \"7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8\") " pod="openshift-insights/insights-runtime-extractor-2w28f" Jul 10 02:11:49.599375 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:49.599316 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8-crio-socket\") pod \"insights-runtime-extractor-2w28f\" (UID: \"7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8\") " pod="openshift-insights/insights-runtime-extractor-2w28f" Jul 10 02:11:49.599375 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:49.599357 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2w28f\" (UID: \"7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8\") " pod="openshift-insights/insights-runtime-extractor-2w28f" Jul 10 02:11:49.599550 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:11:49.599385 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 02:11:49.599550 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:49.599422 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8-crio-socket\") pod \"insights-runtime-extractor-2w28f\" (UID: \"7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8\") " pod="openshift-insights/insights-runtime-extractor-2w28f" Jul 10 02:11:49.599550 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:11:49.599447 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8-kube-rbac-proxy-cm podName:7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8 nodeName:}" failed. No retries permitted until 2026-07-10 02:11:50.099427717 +0000 UTC m=+241.675704798 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2w28f" (UID: "7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8") : configmap "kube-rbac-proxy" not found Jul 10 02:11:49.599550 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:11:49.599468 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 02:11:49.599550 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:11:49.599524 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8-insights-runtime-extractor-tls podName:7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8 nodeName:}" failed. No retries permitted until 2026-07-10 02:11:50.099508666 +0000 UTC m=+241.675785747 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2w28f" (UID: "7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8") : secret "insights-runtime-extractor-tls" not found Jul 10 02:11:49.599773 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:49.599671 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8-data-volume\") pod \"insights-runtime-extractor-2w28f\" (UID: \"7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8\") " pod="openshift-insights/insights-runtime-extractor-2w28f" Jul 10 02:11:49.610973 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:49.610944 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jclj2\" (UniqueName: \"kubernetes.io/projected/7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8-kube-api-access-jclj2\") pod \"insights-runtime-extractor-2w28f\" (UID: \"7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8\") " pod="openshift-insights/insights-runtime-extractor-2w28f" Jul 10 02:11:49.743325 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:49.743287 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:11:49.752374 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:49.752318 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-gsh8k"] Jul 10 02:11:49.754364 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:49.754349 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-gsh8k" Jul 10 02:11:49.757290 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:49.757271 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 10 02:11:49.757404 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:49.757336 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-qfttd\"" Jul 10 02:11:49.757404 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:49.757337 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 10 02:11:49.764942 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:49.764922 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-gsh8k"] Jul 10 02:11:49.838051 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:49.838030 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:11:49.901945 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:49.901879 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjwpp\" (UniqueName: \"kubernetes.io/projected/c3de8f05-0340-4da3-b0a6-66534889f709-kube-api-access-xjwpp\") pod \"migrator-7f59b54f6c-gsh8k\" (UID: \"c3de8f05-0340-4da3-b0a6-66534889f709\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-gsh8k" Jul 10 02:11:49.939892 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:49.939815 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:11:50.003227 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:50.003196 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjwpp\" (UniqueName: \"kubernetes.io/projected/c3de8f05-0340-4da3-b0a6-66534889f709-kube-api-access-xjwpp\") pod \"migrator-7f59b54f6c-gsh8k\" (UID: \"c3de8f05-0340-4da3-b0a6-66534889f709\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-gsh8k" Jul 10 02:11:50.013235 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:50.013214 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjwpp\" (UniqueName: \"kubernetes.io/projected/c3de8f05-0340-4da3-b0a6-66534889f709-kube-api-access-xjwpp\") pod \"migrator-7f59b54f6c-gsh8k\" (UID: \"c3de8f05-0340-4da3-b0a6-66534889f709\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-gsh8k" Jul 10 02:11:50.040462 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:50.040437 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:11:50.063546 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:50.063526 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-gsh8k" Jul 10 02:11:50.104592 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:50.104560 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2w28f\" (UID: \"7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8\") " pod="openshift-insights/insights-runtime-extractor-2w28f" Jul 10 02:11:50.104731 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:50.104646 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2w28f\" (UID: \"7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8\") " pod="openshift-insights/insights-runtime-extractor-2w28f" Jul 10 02:11:50.104731 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:11:50.104679 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 02:11:50.104799 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:11:50.104742 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 02:11:50.104799 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:11:50.104744 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8-kube-rbac-proxy-cm podName:7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8 nodeName:}" failed. No retries permitted until 2026-07-10 02:11:51.104728096 +0000 UTC m=+242.681005160 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2w28f" (UID: "7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8") : configmap "kube-rbac-proxy" not found Jul 10 02:11:50.104799 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:11:50.104794 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8-insights-runtime-extractor-tls podName:7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8 nodeName:}" failed. No retries permitted until 2026-07-10 02:11:51.104779163 +0000 UTC m=+242.681056230 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2w28f" (UID: "7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8") : secret "insights-runtime-extractor-tls" not found Jul 10 02:11:50.138861 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:50.138152 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:11:50.184101 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:50.184028 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-gsh8k"] Jul 10 02:11:50.186700 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:11:50.186671 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3de8f05_0340_4da3_b0a6_66534889f709.slice/crio-91b39d0277619612fb1445efb1bee2e468533c3e856779745cad1b044f49f1d9 WatchSource:0}: Error finding container 91b39d0277619612fb1445efb1bee2e468533c3e856779745cad1b044f49f1d9: Status 404 returned error can't find the container with id 91b39d0277619612fb1445efb1bee2e468533c3e856779745cad1b044f49f1d9 Jul 10 02:11:50.488303 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:50.488220 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:11:50.588847 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:50.588807 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:11:50.659439 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:50.659410 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-gsh8k" event={"ID":"c3de8f05-0340-4da3-b0a6-66534889f709","Type":"ContainerStarted","Data":"91b39d0277619612fb1445efb1bee2e468533c3e856779745cad1b044f49f1d9"} Jul 10 02:11:51.111691 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:51.111657 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2w28f\" (UID: \"7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8\") " pod="openshift-insights/insights-runtime-extractor-2w28f" Jul 10 02:11:51.112137 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:51.111727 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2w28f\" (UID: \"7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8\") " pod="openshift-insights/insights-runtime-extractor-2w28f" Jul 10 02:11:51.112137 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:11:51.111806 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 02:11:51.112137 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:11:51.111820 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 02:11:51.112137 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:11:51.111878 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8-insights-runtime-extractor-tls podName:7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8 nodeName:}" failed. No retries permitted until 2026-07-10 02:11:53.111862956 +0000 UTC m=+244.688140038 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2w28f" (UID: "7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8") : secret "insights-runtime-extractor-tls" not found Jul 10 02:11:51.112137 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:11:51.111894 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8-kube-rbac-proxy-cm podName:7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8 nodeName:}" failed. No retries permitted until 2026-07-10 02:11:53.111887903 +0000 UTC m=+244.688164980 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2w28f" (UID: "7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8") : configmap "kube-rbac-proxy" not found Jul 10 02:11:51.621517 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:51.621448 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xgndm_8ccffaf3-b29f-4f58-ba37-5ab6f4992d19/dns-node-resolver/0.log" Jul 10 02:11:51.663792 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:51.663755 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-gsh8k" event={"ID":"c3de8f05-0340-4da3-b0a6-66534889f709","Type":"ContainerStarted","Data":"cc6e1908488b36707868311149aa8b67aacdb62c27369369cc6dfb2bfad9fbe8"} Jul 10 02:11:51.663935 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:51.663830 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-gsh8k" event={"ID":"c3de8f05-0340-4da3-b0a6-66534889f709","Type":"ContainerStarted","Data":"808bc6c141e56d05dec3c81c94679a9c3ed1ebfc0a3345c8be42256f7a241547"} Jul 10 02:11:51.681484 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:51.681436 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-gsh8k" podStartSLOduration=1.510214898 podStartE2EDuration="2.681421675s" totalImagesPullingTime="1.171206777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:11:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:11:51.68027891 +0000 UTC m=+243.256555997" watchObservedRunningTime="2026-07-10 02:11:51.681421675 +0000 UTC m=+243.257698795" Jul 10 02:11:51.762568 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:51.762519 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:11:52.085920 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:52.085892 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-g7dx9"] Jul 10 02:11:52.087983 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:52.087968 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-g7dx9" Jul 10 02:11:52.090866 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:52.090846 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 10 02:11:52.090971 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:52.090871 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 10 02:11:52.090971 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:52.090899 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 10 02:11:52.090971 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:52.090847 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 10 02:11:52.092034 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:52.092019 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-8b2z2\"" Jul 10 02:11:52.097047 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:52.097030 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-g7dx9"] Jul 10 02:11:52.192642 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:52.192592 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:11:52.222639 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:52.222592 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/60819479-a44f-4e23-8db4-51a1901aa6a3-signing-key\") pod \"service-ca-79f68b9f4f-g7dx9\" (UID: \"60819479-a44f-4e23-8db4-51a1901aa6a3\") " pod="openshift-service-ca/service-ca-79f68b9f4f-g7dx9" Jul 10 02:11:52.222760 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:52.222745 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/60819479-a44f-4e23-8db4-51a1901aa6a3-signing-cabundle\") pod \"service-ca-79f68b9f4f-g7dx9\" (UID: \"60819479-a44f-4e23-8db4-51a1901aa6a3\") " pod="openshift-service-ca/service-ca-79f68b9f4f-g7dx9" Jul 10 02:11:52.222801 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:52.222775 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg55n\" (UniqueName: \"kubernetes.io/projected/60819479-a44f-4e23-8db4-51a1901aa6a3-kube-api-access-bg55n\") pod \"service-ca-79f68b9f4f-g7dx9\" (UID: \"60819479-a44f-4e23-8db4-51a1901aa6a3\") " pod="openshift-service-ca/service-ca-79f68b9f4f-g7dx9" Jul 10 02:11:52.295890 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:52.295860 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:11:52.323995 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:52.323964 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/60819479-a44f-4e23-8db4-51a1901aa6a3-signing-cabundle\") pod \"service-ca-79f68b9f4f-g7dx9\" (UID: \"60819479-a44f-4e23-8db4-51a1901aa6a3\") " pod="openshift-service-ca/service-ca-79f68b9f4f-g7dx9" Jul 10 02:11:52.324069 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:52.324004 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg55n\" (UniqueName: \"kubernetes.io/projected/60819479-a44f-4e23-8db4-51a1901aa6a3-kube-api-access-bg55n\") pod \"service-ca-79f68b9f4f-g7dx9\" (UID: \"60819479-a44f-4e23-8db4-51a1901aa6a3\") " pod="openshift-service-ca/service-ca-79f68b9f4f-g7dx9" Jul 10 02:11:52.324069 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:52.324055 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/60819479-a44f-4e23-8db4-51a1901aa6a3-signing-key\") pod \"service-ca-79f68b9f4f-g7dx9\" (UID: \"60819479-a44f-4e23-8db4-51a1901aa6a3\") " pod="openshift-service-ca/service-ca-79f68b9f4f-g7dx9" Jul 10 02:11:52.324615 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:52.324574 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/60819479-a44f-4e23-8db4-51a1901aa6a3-signing-cabundle\") pod \"service-ca-79f68b9f4f-g7dx9\" (UID: \"60819479-a44f-4e23-8db4-51a1901aa6a3\") " pod="openshift-service-ca/service-ca-79f68b9f4f-g7dx9" Jul 10 02:11:52.326366 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:52.326347 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/60819479-a44f-4e23-8db4-51a1901aa6a3-signing-key\") pod \"service-ca-79f68b9f4f-g7dx9\" (UID: \"60819479-a44f-4e23-8db4-51a1901aa6a3\") " pod="openshift-service-ca/service-ca-79f68b9f4f-g7dx9" Jul 10 02:11:52.333277 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:52.333253 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg55n\" (UniqueName: \"kubernetes.io/projected/60819479-a44f-4e23-8db4-51a1901aa6a3-kube-api-access-bg55n\") pod \"service-ca-79f68b9f4f-g7dx9\" (UID: \"60819479-a44f-4e23-8db4-51a1901aa6a3\") " pod="openshift-service-ca/service-ca-79f68b9f4f-g7dx9" Jul 10 02:11:52.396380 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:52.396331 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-g7dx9" Jul 10 02:11:52.421933 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:52.421908 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8t46j_86176d1c-3214-46c1-91fe-94f087e0bda4/node-ca/0.log" Jul 10 02:11:52.515542 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:52.515485 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-g7dx9"] Jul 10 02:11:52.518045 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:11:52.518020 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60819479_a44f_4e23_8db4_51a1901aa6a3.slice/crio-7cd134ee0daae5bb41ed89d0832370ea7bc9b195108e647fb30e8ae095e7cba9 WatchSource:0}: Error finding container 7cd134ee0daae5bb41ed89d0832370ea7bc9b195108e647fb30e8ae095e7cba9: Status 404 returned error can't find the container with id 7cd134ee0daae5bb41ed89d0832370ea7bc9b195108e647fb30e8ae095e7cba9 Jul 10 02:11:52.668193 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:52.668114 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-g7dx9" event={"ID":"60819479-a44f-4e23-8db4-51a1901aa6a3","Type":"ContainerStarted","Data":"36a0524ab75fa8012158cc8250bc898b142faf89376ce76e65ccd60714176ed7"} Jul 10 02:11:52.668193 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:52.668164 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-g7dx9" event={"ID":"60819479-a44f-4e23-8db4-51a1901aa6a3","Type":"ContainerStarted","Data":"7cd134ee0daae5bb41ed89d0832370ea7bc9b195108e647fb30e8ae095e7cba9"} Jul 10 02:11:52.691241 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:52.691194 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-g7dx9" podStartSLOduration=0.691179889 podStartE2EDuration="691.179889ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:11:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:11:52.690237401 +0000 UTC m=+244.266514490" watchObservedRunningTime="2026-07-10 02:11:52.691179889 +0000 UTC m=+244.267456977" Jul 10 02:11:52.930331 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:52.930244 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82ce150d-af95-4650-a996-fbe49696196f-metrics-certs\") pod \"router-default-6cd68c6448-czfxh\" (UID: \"82ce150d-af95-4650-a996-fbe49696196f\") " pod="openshift-ingress/router-default-6cd68c6448-czfxh" Jul 10 02:11:52.930331 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:52.930282 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82ce150d-af95-4650-a996-fbe49696196f-service-ca-bundle\") pod \"router-default-6cd68c6448-czfxh\" (UID: \"82ce150d-af95-4650-a996-fbe49696196f\") " pod="openshift-ingress/router-default-6cd68c6448-czfxh" Jul 10 02:11:52.930520 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:52.930363 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ebeadbc-c34a-4aae-aeab-27ea702d6cf3-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-8ghxz\" (UID: \"2ebeadbc-c34a-4aae-aeab-27ea702d6cf3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-8ghxz" Jul 10 02:11:52.930520 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:11:52.930397 2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 02:11:52.930520 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:11:52.930462 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82ce150d-af95-4650-a996-fbe49696196f-metrics-certs podName:82ce150d-af95-4650-a996-fbe49696196f nodeName:}" failed. No retries permitted until 2026-07-10 02:12:00.930443531 +0000 UTC m=+252.506720606 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/82ce150d-af95-4650-a996-fbe49696196f-metrics-certs") pod "router-default-6cd68c6448-czfxh" (UID: "82ce150d-af95-4650-a996-fbe49696196f") : secret "router-metrics-certs-default" not found Jul 10 02:11:52.930520 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:11:52.930484 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82ce150d-af95-4650-a996-fbe49696196f-service-ca-bundle podName:82ce150d-af95-4650-a996-fbe49696196f nodeName:}" failed. No retries permitted until 2026-07-10 02:12:00.930468854 +0000 UTC m=+252.506745930 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/82ce150d-af95-4650-a996-fbe49696196f-service-ca-bundle") pod "router-default-6cd68c6448-czfxh" (UID: "82ce150d-af95-4650-a996-fbe49696196f") : configmap references non-existent config key: service-ca.crt Jul 10 02:11:52.930520 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:11:52.930487 2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 02:11:52.930723 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:11:52.930528 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ebeadbc-c34a-4aae-aeab-27ea702d6cf3-samples-operator-tls podName:2ebeadbc-c34a-4aae-aeab-27ea702d6cf3 nodeName:}" failed. No retries permitted until 2026-07-10 02:12:00.93051737 +0000 UTC m=+252.506794455 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2ebeadbc-c34a-4aae-aeab-27ea702d6cf3-samples-operator-tls") pod "cluster-samples-operator-65db858894-8ghxz" (UID: "2ebeadbc-c34a-4aae-aeab-27ea702d6cf3") : secret "samples-operator-tls" not found Jul 10 02:11:53.042165 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:53.042127 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:11:53.132126 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:53.132101 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2w28f\" (UID: \"7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8\") " pod="openshift-insights/insights-runtime-extractor-2w28f" Jul 10 02:11:53.132289 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:53.132167 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2w28f\" (UID: \"7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8\") " pod="openshift-insights/insights-runtime-extractor-2w28f" Jul 10 02:11:53.132289 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:11:53.132228 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 02:11:53.132289 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:11:53.132264 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 02:11:53.132289 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:11:53.132278 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8-kube-rbac-proxy-cm podName:7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8 nodeName:}" failed. No retries permitted until 2026-07-10 02:11:57.132263957 +0000 UTC m=+248.708541021 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2w28f" (UID: "7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8") : configmap "kube-rbac-proxy" not found Jul 10 02:11:53.132435 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:11:53.132315 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8-insights-runtime-extractor-tls podName:7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8 nodeName:}" failed. No retries permitted until 2026-07-10 02:11:57.132302487 +0000 UTC m=+248.708579551 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2w28f" (UID: "7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8") : secret "insights-runtime-extractor-tls" not found Jul 10 02:11:53.222083 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:53.222004 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-r7nm4_2ef36e20-1fdf-4327-bf59-80abf4dfb7bd/serve-healthcheck-canary/0.log" Jul 10 02:11:54.165302 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:54.165266 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:11:54.387863 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:54.387830 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:11:54.532868 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:54.532835 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:11:54.636194 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:54.636163 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:11:54.733551 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:54.733519 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:11:54.862484 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:54.862398 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:11:55.629027 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:55.628995 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:11:56.057372 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:56.057341 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:11:56.258138 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:56.258105 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:11:56.453786 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:56.453704 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:11:56.656744 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:56.656709 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:11:56.851223 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:56.851187 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:11:57.166351 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:57.166267 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2w28f\" (UID: \"7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8\") " pod="openshift-insights/insights-runtime-extractor-2w28f" Jul 10 02:11:57.166481 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:57.166354 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2w28f\" (UID: \"7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8\") " pod="openshift-insights/insights-runtime-extractor-2w28f" Jul 10 02:11:57.166481 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:11:57.166421 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 02:11:57.166481 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:11:57.166425 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 02:11:57.166481 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:11:57.166475 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8-kube-rbac-proxy-cm podName:7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8 nodeName:}" failed. No retries permitted until 2026-07-10 02:12:05.166461902 +0000 UTC m=+256.742738967 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2w28f" (UID: "7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8") : configmap "kube-rbac-proxy" not found Jul 10 02:11:57.166649 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:11:57.166487 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8-insights-runtime-extractor-tls podName:7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8 nodeName:}" failed. No retries permitted until 2026-07-10 02:12:05.166481464 +0000 UTC m=+256.742758527 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2w28f" (UID: "7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8") : secret "insights-runtime-extractor-tls" not found Jul 10 02:11:57.882772 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:57.882740 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:11:57.943735 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:57.943702 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:11:58.074711 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:58.074680 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:11:58.251831 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:58.251762 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:11:58.457785 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:58.457747 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:11:58.654450 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:58.654411 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:11:58.851167 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:58.851139 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:11:59.007097 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:59.007001 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:11:59.858970 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:11:59.858936 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:00.250781 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:00.250693 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:00.453914 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:00.453882 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:00.554989 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:00.554952 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:00.655522 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:00.655486 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:00.793103 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:00.793069 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bf086af-02b9-43f2-935f-474133fef4cf-metrics-certs\") pod \"network-metrics-daemon-6hfgs\" (UID: \"8bf086af-02b9-43f2-935f-474133fef4cf\") " pod="openshift-multus/network-metrics-daemon-6hfgs" Jul 10 02:12:00.793268 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:12:00.793219 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 10 02:12:00.793306 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:12:00.793280 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bf086af-02b9-43f2-935f-474133fef4cf-metrics-certs podName:8bf086af-02b9-43f2-935f-474133fef4cf nodeName:}" failed. No retries permitted until 2026-07-10 02:14:02.793263017 +0000 UTC m=+374.369540081 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8bf086af-02b9-43f2-935f-474133fef4cf-metrics-certs") pod "network-metrics-daemon-6hfgs" (UID: "8bf086af-02b9-43f2-935f-474133fef4cf") : secret "metrics-daemon-secret" not found Jul 10 02:12:00.850492 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:00.850408 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:00.995211 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:00.995176 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82ce150d-af95-4650-a996-fbe49696196f-metrics-certs\") pod \"router-default-6cd68c6448-czfxh\" (UID: \"82ce150d-af95-4650-a996-fbe49696196f\") " pod="openshift-ingress/router-default-6cd68c6448-czfxh" Jul 10 02:12:00.995363 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:00.995220 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82ce150d-af95-4650-a996-fbe49696196f-service-ca-bundle\") pod \"router-default-6cd68c6448-czfxh\" (UID: \"82ce150d-af95-4650-a996-fbe49696196f\") " pod="openshift-ingress/router-default-6cd68c6448-czfxh" Jul 10 02:12:00.995444 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:00.995391 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ebeadbc-c34a-4aae-aeab-27ea702d6cf3-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-8ghxz\" (UID: \"2ebeadbc-c34a-4aae-aeab-27ea702d6cf3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-8ghxz" Jul 10 02:12:00.995912 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:00.995888 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82ce150d-af95-4650-a996-fbe49696196f-service-ca-bundle\") pod \"router-default-6cd68c6448-czfxh\" (UID: \"82ce150d-af95-4650-a996-fbe49696196f\") " pod="openshift-ingress/router-default-6cd68c6448-czfxh" Jul 10 02:12:00.997619 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:00.997584 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ebeadbc-c34a-4aae-aeab-27ea702d6cf3-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-8ghxz\" (UID: \"2ebeadbc-c34a-4aae-aeab-27ea702d6cf3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-8ghxz" Jul 10 02:12:00.997716 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:00.997655 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82ce150d-af95-4650-a996-fbe49696196f-metrics-certs\") pod \"router-default-6cd68c6448-czfxh\" (UID: \"82ce150d-af95-4650-a996-fbe49696196f\") " pod="openshift-ingress/router-default-6cd68c6448-czfxh" Jul 10 02:12:01.117199 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:01.117112 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-pfdvp\"" Jul 10 02:12:01.124812 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:01.124794 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-8ghxz" Jul 10 02:12:01.128450 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:01.128429 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-frwtg\"" Jul 10 02:12:01.137032 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:01.136884 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6cd68c6448-czfxh" Jul 10 02:12:01.255426 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:01.255391 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-8ghxz"] Jul 10 02:12:01.273463 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:01.273441 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6cd68c6448-czfxh"] Jul 10 02:12:01.275126 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:12:01.275102 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82ce150d_af95_4650_a996_fbe49696196f.slice/crio-abc37dbf1747b6792beb8ad2af14964d12b0059ebdcb1b57d932d5675368503a WatchSource:0}: Error finding container abc37dbf1747b6792beb8ad2af14964d12b0059ebdcb1b57d932d5675368503a: Status 404 returned error can't find the container with id abc37dbf1747b6792beb8ad2af14964d12b0059ebdcb1b57d932d5675368503a Jul 10 02:12:01.705075 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:01.705038 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6cd68c6448-czfxh" event={"ID":"82ce150d-af95-4650-a996-fbe49696196f","Type":"ContainerStarted","Data":"de535ff0743f11d52768b13f1247d402d42c7870ed5ebf164d160e392e5c5774"} Jul 10 02:12:01.705241 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:01.705086 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6cd68c6448-czfxh" event={"ID":"82ce150d-af95-4650-a996-fbe49696196f","Type":"ContainerStarted","Data":"abc37dbf1747b6792beb8ad2af14964d12b0059ebdcb1b57d932d5675368503a"} Jul 10 02:12:01.706162 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:01.706131 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-8ghxz" event={"ID":"2ebeadbc-c34a-4aae-aeab-27ea702d6cf3","Type":"ContainerStarted","Data":"e80b00d6c0a60ae883e64570e55658fd85bd01dd5abdbac290e64598656e85e3"} Jul 10 02:12:01.727821 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:01.727782 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6cd68c6448-czfxh" podStartSLOduration=16.727767917 podStartE2EDuration="16.727767917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:11:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:12:01.726784513 +0000 UTC m=+253.303061599" watchObservedRunningTime="2026-07-10 02:12:01.727767917 +0000 UTC m=+253.304045043" Jul 10 02:12:01.854045 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:01.853956 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:02.138458 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:02.137637 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6cd68c6448-czfxh" Jul 10 02:12:02.144329 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:02.144309 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6cd68c6448-czfxh" Jul 10 02:12:02.260324 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:02.260289 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:02.457301 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:02.457209 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:02.591166 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:02.591092 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:02.692396 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:02.692347 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:02.709681 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:02.709614 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6cd68c6448-czfxh" Jul 10 02:12:02.710915 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:02.710897 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6cd68c6448-czfxh" Jul 10 02:12:02.865021 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:02.864989 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:03.714570 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:03.714533 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-8ghxz" event={"ID":"2ebeadbc-c34a-4aae-aeab-27ea702d6cf3","Type":"ContainerStarted","Data":"3ffe8fdc3049cb2872edcf90c0035451a45a527bd76e51e2bf55e6217b2e0048"} Jul 10 02:12:03.714977 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:03.714589 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-8ghxz" event={"ID":"2ebeadbc-c34a-4aae-aeab-27ea702d6cf3","Type":"ContainerStarted","Data":"dbceb9114f3ac7f3d9661c142d02df08fcf56cdaa4737f2d23c8129fed154b47"} Jul 10 02:12:03.732766 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:03.732723 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-8ghxz" podStartSLOduration=16.874102412 podStartE2EDuration="18.732707301s" totalImagesPullingTime="1.858604889s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:11:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:12:03.731882497 +0000 UTC m=+255.308159584" watchObservedRunningTime="2026-07-10 02:12:03.732707301 +0000 UTC m=+255.308984386" Jul 10 02:12:03.853033 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:03.853004 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:04.055788 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:04.055724 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:04.456347 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:04.456260 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:04.652994 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:04.652965 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:04.857294 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:04.857260 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:05.226019 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:05.225919 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2w28f\" (UID: \"7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8\") " pod="openshift-insights/insights-runtime-extractor-2w28f" Jul 10 02:12:05.226019 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:05.226013 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2w28f\" (UID: \"7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8\") " pod="openshift-insights/insights-runtime-extractor-2w28f" Jul 10 02:12:05.226583 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:05.226563 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2w28f\" (UID: \"7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8\") " pod="openshift-insights/insights-runtime-extractor-2w28f" Jul 10 02:12:05.228254 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:05.228235 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2w28f\" (UID: \"7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8\") " pod="openshift-insights/insights-runtime-extractor-2w28f" Jul 10 02:12:05.256159 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:05.256139 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2w28f" Jul 10 02:12:05.379669 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:05.379642 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2w28f"] Jul 10 02:12:05.383203 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:12:05.383180 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a7d4a6f_04a7_4c56_b1b6_fe9d77f9a2a8.slice/crio-a50f854fb11245b1b55c7a053ca89c9f8fd14f24ff5d194698c173563b9c206c WatchSource:0}: Error finding container a50f854fb11245b1b55c7a053ca89c9f8fd14f24ff5d194698c173563b9c206c: Status 404 returned error can't find the container with id a50f854fb11245b1b55c7a053ca89c9f8fd14f24ff5d194698c173563b9c206c Jul 10 02:12:05.721858 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:05.721821 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2w28f" event={"ID":"7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8","Type":"ContainerStarted","Data":"a50f854fb11245b1b55c7a053ca89c9f8fd14f24ff5d194698c173563b9c206c"} Jul 10 02:12:08.229714 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:08.229672 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:08.288537 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:08.288499 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:09.737348 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:09.737319 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2w28f" event={"ID":"7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8","Type":"ContainerStarted","Data":"c9077e00e78862ea6e7f81970d43fcda603148bd80fad3e2b84d38fe27f6ac80"} Jul 10 02:12:10.743080 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:10.743000 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2w28f" event={"ID":"7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8","Type":"ContainerStarted","Data":"619d042dda4a60aef9948804d77db9b513e39ab83d68721c0ccb7d88839f7d15"} Jul 10 02:12:12.612504 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:12.612475 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:12.676801 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:12.676778 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:12.751302 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:12.751237 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2w28f" event={"ID":"7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8","Type":"ContainerStarted","Data":"ccf2dd76bff456cc936fd04ae0e5b889a62b2c4080c949078535c54d0edfd737"} Jul 10 02:12:12.770974 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:12.770931 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2w28f" podStartSLOduration=17.10068554 podStartE2EDuration="23.770912597s" totalImagesPullingTime="6.670227057s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 02:11:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:12:12.770907442 +0000 UTC m=+264.347184527" watchObservedRunningTime="2026-07-10 02:12:12.770912597 +0000 UTC m=+264.347189682" Jul 10 02:12:15.318718 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:15.318683 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:15.385748 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:15.385718 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:19.485291 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:19.485257 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:19.588042 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:19.588016 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:19.653958 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:19.653924 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:19.870981 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:19.870944 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:21.677714 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:21.677662 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:21.748955 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:21.748917 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:22.170049 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:22.170017 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:24.284377 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:24.284336 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:25.819647 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:25.819613 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:26.035235 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.035151 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:26.088403 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.088318 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-tplmw"] Jul 10 02:12:26.091243 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.091221 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tplmw" Jul 10 02:12:26.094138 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.094120 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 10 02:12:26.094520 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.094470 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 10 02:12:26.094724 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.094526 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-6w2zn\"" Jul 10 02:12:26.094724 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.094645 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 10 02:12:26.095936 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.095913 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 10 02:12:26.096033 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.095916 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 10 02:12:26.112818 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.112768 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-t64f9"] Jul 10 02:12:26.115387 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.115368 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-tplmw"] Jul 10 02:12:26.115526 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.115510 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-t64f9" Jul 10 02:12:26.119055 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.118827 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 10 02:12:26.119055 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.118845 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 10 02:12:26.119211 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.119112 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hg988\"" Jul 10 02:12:26.119211 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.119156 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 10 02:12:26.128768 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.128746 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:26.137303 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.137284 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-nznmv"] Jul 10 02:12:26.140117 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.140100 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-nznmv" Jul 10 02:12:26.142863 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.142844 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 10 02:12:26.143008 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.142992 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 10 02:12:26.143008 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.143005 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 10 02:12:26.143134 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.143056 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-fgvbw\"" Jul 10 02:12:26.149464 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.149418 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-nznmv"] Jul 10 02:12:26.189283 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.189246 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5d54\" (UniqueName: \"kubernetes.io/projected/4642e40c-4306-467a-b96a-87311adc28e5-kube-api-access-w5d54\") pod \"kube-state-metrics-785778ff74-nznmv\" (UID: \"4642e40c-4306-467a-b96a-87311adc28e5\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nznmv" Jul 10 02:12:26.189466 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.189292 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef076029-9853-4835-b6ac-ee0a295c23fc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-tplmw\" (UID: \"ef076029-9853-4835-b6ac-ee0a295c23fc\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tplmw" Jul 10 02:12:26.189466 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.189323 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4642e40c-4306-467a-b96a-87311adc28e5-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-nznmv\" (UID: \"4642e40c-4306-467a-b96a-87311adc28e5\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nznmv" Jul 10 02:12:26.189466 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.189388 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4642e40c-4306-467a-b96a-87311adc28e5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-nznmv\" (UID: \"4642e40c-4306-467a-b96a-87311adc28e5\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nznmv" Jul 10 02:12:26.189466 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.189455 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ef076029-9853-4835-b6ac-ee0a295c23fc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-tplmw\" (UID: \"ef076029-9853-4835-b6ac-ee0a295c23fc\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tplmw" Jul 10 02:12:26.189685 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.189480 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fa530c6a-9f5d-4919-bb59-f3e7752a6c79-root\") pod \"node-exporter-t64f9\" (UID: \"fa530c6a-9f5d-4919-bb59-f3e7752a6c79\") " pod="openshift-monitoring/node-exporter-t64f9" Jul 10 02:12:26.189685 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.189530 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fa530c6a-9f5d-4919-bb59-f3e7752a6c79-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t64f9\" (UID: \"fa530c6a-9f5d-4919-bb59-f3e7752a6c79\") " pod="openshift-monitoring/node-exporter-t64f9" Jul 10 02:12:26.189685 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.189585 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef076029-9853-4835-b6ac-ee0a295c23fc-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-tplmw\" (UID: \"ef076029-9853-4835-b6ac-ee0a295c23fc\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tplmw" Jul 10 02:12:26.189685 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.189626 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fa530c6a-9f5d-4919-bb59-f3e7752a6c79-node-exporter-wtmp\") pod \"node-exporter-t64f9\" (UID: \"fa530c6a-9f5d-4919-bb59-f3e7752a6c79\") " pod="openshift-monitoring/node-exporter-t64f9" Jul 10 02:12:26.189685 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.189654 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa530c6a-9f5d-4919-bb59-f3e7752a6c79-metrics-client-ca\") pod \"node-exporter-t64f9\" (UID: \"fa530c6a-9f5d-4919-bb59-f3e7752a6c79\") " pod="openshift-monitoring/node-exporter-t64f9" Jul 10 02:12:26.189917 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.189692 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj529\" (UniqueName: \"kubernetes.io/projected/fa530c6a-9f5d-4919-bb59-f3e7752a6c79-kube-api-access-xj529\") pod \"node-exporter-t64f9\" (UID: \"fa530c6a-9f5d-4919-bb59-f3e7752a6c79\") " pod="openshift-monitoring/node-exporter-t64f9" Jul 10 02:12:26.189917 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.189716 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4642e40c-4306-467a-b96a-87311adc28e5-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-nznmv\" (UID: \"4642e40c-4306-467a-b96a-87311adc28e5\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nznmv" Jul 10 02:12:26.189917 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.189742 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fa530c6a-9f5d-4919-bb59-f3e7752a6c79-node-exporter-tls\") pod \"node-exporter-t64f9\" (UID: \"fa530c6a-9f5d-4919-bb59-f3e7752a6c79\") " pod="openshift-monitoring/node-exporter-t64f9" Jul 10 02:12:26.189917 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.189771 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4642e40c-4306-467a-b96a-87311adc28e5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-nznmv\" (UID: \"4642e40c-4306-467a-b96a-87311adc28e5\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nznmv" Jul 10 02:12:26.189917 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.189801 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fa530c6a-9f5d-4919-bb59-f3e7752a6c79-node-exporter-accelerators-collector-config\") pod \"node-exporter-t64f9\" (UID: \"fa530c6a-9f5d-4919-bb59-f3e7752a6c79\") " pod="openshift-monitoring/node-exporter-t64f9" Jul 10 02:12:26.189917 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.189841 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4642e40c-4306-467a-b96a-87311adc28e5-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-nznmv\" (UID: \"4642e40c-4306-467a-b96a-87311adc28e5\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nznmv" Jul 10 02:12:26.190215 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.189925 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fa530c6a-9f5d-4919-bb59-f3e7752a6c79-sys\") pod \"node-exporter-t64f9\" (UID: \"fa530c6a-9f5d-4919-bb59-f3e7752a6c79\") " pod="openshift-monitoring/node-exporter-t64f9" Jul 10 02:12:26.190215 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.189988 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck2sx\" (UniqueName: \"kubernetes.io/projected/ef076029-9853-4835-b6ac-ee0a295c23fc-kube-api-access-ck2sx\") pod \"openshift-state-metrics-84bd68b896-tplmw\" (UID: \"ef076029-9853-4835-b6ac-ee0a295c23fc\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tplmw" Jul 10 02:12:26.190215 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.190041 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fa530c6a-9f5d-4919-bb59-f3e7752a6c79-node-exporter-textfile\") pod \"node-exporter-t64f9\" (UID: \"fa530c6a-9f5d-4919-bb59-f3e7752a6c79\") " pod="openshift-monitoring/node-exporter-t64f9" Jul 10 02:12:26.218867 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.218832 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:26.290867 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.290829 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa530c6a-9f5d-4919-bb59-f3e7752a6c79-metrics-client-ca\") pod \"node-exporter-t64f9\" (UID: \"fa530c6a-9f5d-4919-bb59-f3e7752a6c79\") " pod="openshift-monitoring/node-exporter-t64f9" Jul 10 02:12:26.290867 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.290864 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj529\" (UniqueName: \"kubernetes.io/projected/fa530c6a-9f5d-4919-bb59-f3e7752a6c79-kube-api-access-xj529\") pod \"node-exporter-t64f9\" (UID: \"fa530c6a-9f5d-4919-bb59-f3e7752a6c79\") " pod="openshift-monitoring/node-exporter-t64f9" Jul 10 02:12:26.291098 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.290888 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4642e40c-4306-467a-b96a-87311adc28e5-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-nznmv\" (UID: \"4642e40c-4306-467a-b96a-87311adc28e5\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nznmv" Jul 10 02:12:26.291098 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.290946 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fa530c6a-9f5d-4919-bb59-f3e7752a6c79-node-exporter-tls\") pod \"node-exporter-t64f9\" (UID: \"fa530c6a-9f5d-4919-bb59-f3e7752a6c79\") " pod="openshift-monitoring/node-exporter-t64f9" Jul 10 02:12:26.291098 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.290977 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4642e40c-4306-467a-b96a-87311adc28e5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-nznmv\" (UID: \"4642e40c-4306-467a-b96a-87311adc28e5\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nznmv" Jul 10 02:12:26.291098 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.291001 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fa530c6a-9f5d-4919-bb59-f3e7752a6c79-node-exporter-accelerators-collector-config\") pod \"node-exporter-t64f9\" (UID: \"fa530c6a-9f5d-4919-bb59-f3e7752a6c79\") " pod="openshift-monitoring/node-exporter-t64f9" Jul 10 02:12:26.291098 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.291027 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4642e40c-4306-467a-b96a-87311adc28e5-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-nznmv\" (UID: \"4642e40c-4306-467a-b96a-87311adc28e5\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nznmv" Jul 10 02:12:26.291098 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.291064 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fa530c6a-9f5d-4919-bb59-f3e7752a6c79-sys\") pod \"node-exporter-t64f9\" (UID: \"fa530c6a-9f5d-4919-bb59-f3e7752a6c79\") " pod="openshift-monitoring/node-exporter-t64f9" Jul 10 02:12:26.291372 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:12:26.291102 2566 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 10 02:12:26.291372 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.291112 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck2sx\" (UniqueName: \"kubernetes.io/projected/ef076029-9853-4835-b6ac-ee0a295c23fc-kube-api-access-ck2sx\") pod \"openshift-state-metrics-84bd68b896-tplmw\" (UID: \"ef076029-9853-4835-b6ac-ee0a295c23fc\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tplmw" Jul 10 02:12:26.291372 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:12:26.291173 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa530c6a-9f5d-4919-bb59-f3e7752a6c79-node-exporter-tls podName:fa530c6a-9f5d-4919-bb59-f3e7752a6c79 nodeName:}" failed. No retries permitted until 2026-07-10 02:12:26.791154898 +0000 UTC m=+278.367431982 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/fa530c6a-9f5d-4919-bb59-f3e7752a6c79-node-exporter-tls") pod "node-exporter-t64f9" (UID: "fa530c6a-9f5d-4919-bb59-f3e7752a6c79") : secret "node-exporter-tls" not found Jul 10 02:12:26.291372 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.291191 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fa530c6a-9f5d-4919-bb59-f3e7752a6c79-node-exporter-textfile\") pod \"node-exporter-t64f9\" (UID: \"fa530c6a-9f5d-4919-bb59-f3e7752a6c79\") " pod="openshift-monitoring/node-exporter-t64f9" Jul 10 02:12:26.291372 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.291227 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5d54\" (UniqueName: \"kubernetes.io/projected/4642e40c-4306-467a-b96a-87311adc28e5-kube-api-access-w5d54\") pod \"kube-state-metrics-785778ff74-nznmv\" (UID: \"4642e40c-4306-467a-b96a-87311adc28e5\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nznmv" Jul 10 02:12:26.291372 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.291260 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef076029-9853-4835-b6ac-ee0a295c23fc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-tplmw\" (UID: \"ef076029-9853-4835-b6ac-ee0a295c23fc\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tplmw" Jul 10 02:12:26.291372 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.291289 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4642e40c-4306-467a-b96a-87311adc28e5-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-nznmv\" (UID: \"4642e40c-4306-467a-b96a-87311adc28e5\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nznmv" Jul 10 02:12:26.291372 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.291317 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fa530c6a-9f5d-4919-bb59-f3e7752a6c79-sys\") pod \"node-exporter-t64f9\" (UID: \"fa530c6a-9f5d-4919-bb59-f3e7752a6c79\") " pod="openshift-monitoring/node-exporter-t64f9" Jul 10 02:12:26.291372 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.291321 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4642e40c-4306-467a-b96a-87311adc28e5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-nznmv\" (UID: \"4642e40c-4306-467a-b96a-87311adc28e5\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nznmv" Jul 10 02:12:26.291870 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.291400 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ef076029-9853-4835-b6ac-ee0a295c23fc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-tplmw\" (UID: \"ef076029-9853-4835-b6ac-ee0a295c23fc\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tplmw" Jul 10 02:12:26.291870 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.291431 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fa530c6a-9f5d-4919-bb59-f3e7752a6c79-root\") pod \"node-exporter-t64f9\" (UID: \"fa530c6a-9f5d-4919-bb59-f3e7752a6c79\") " pod="openshift-monitoring/node-exporter-t64f9" Jul 10 02:12:26.291870 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.291460 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fa530c6a-9f5d-4919-bb59-f3e7752a6c79-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t64f9\" (UID: \"fa530c6a-9f5d-4919-bb59-f3e7752a6c79\") " pod="openshift-monitoring/node-exporter-t64f9" Jul 10 02:12:26.291870 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.291492 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef076029-9853-4835-b6ac-ee0a295c23fc-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-tplmw\" (UID: \"ef076029-9853-4835-b6ac-ee0a295c23fc\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tplmw" Jul 10 02:12:26.291870 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.291512 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa530c6a-9f5d-4919-bb59-f3e7752a6c79-metrics-client-ca\") pod \"node-exporter-t64f9\" (UID: \"fa530c6a-9f5d-4919-bb59-f3e7752a6c79\") " pod="openshift-monitoring/node-exporter-t64f9" Jul 10 02:12:26.291870 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.291523 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fa530c6a-9f5d-4919-bb59-f3e7752a6c79-node-exporter-wtmp\") pod \"node-exporter-t64f9\" (UID: \"fa530c6a-9f5d-4919-bb59-f3e7752a6c79\") " pod="openshift-monitoring/node-exporter-t64f9" Jul 10 02:12:26.291870 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.291625 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4642e40c-4306-467a-b96a-87311adc28e5-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-nznmv\" (UID: \"4642e40c-4306-467a-b96a-87311adc28e5\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nznmv" Jul 10 02:12:26.292199 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.292026 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4642e40c-4306-467a-b96a-87311adc28e5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-nznmv\" (UID: \"4642e40c-4306-467a-b96a-87311adc28e5\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nznmv" Jul 10 02:12:26.292199 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.292026 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fa530c6a-9f5d-4919-bb59-f3e7752a6c79-node-exporter-accelerators-collector-config\") pod \"node-exporter-t64f9\" (UID: \"fa530c6a-9f5d-4919-bb59-f3e7752a6c79\") " pod="openshift-monitoring/node-exporter-t64f9" Jul 10 02:12:26.292199 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.292080 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fa530c6a-9f5d-4919-bb59-f3e7752a6c79-node-exporter-textfile\") pod \"node-exporter-t64f9\" (UID: \"fa530c6a-9f5d-4919-bb59-f3e7752a6c79\") " pod="openshift-monitoring/node-exporter-t64f9" Jul 10 02:12:26.292346 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.292247 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fa530c6a-9f5d-4919-bb59-f3e7752a6c79-node-exporter-wtmp\") pod \"node-exporter-t64f9\" (UID: \"fa530c6a-9f5d-4919-bb59-f3e7752a6c79\") " pod="openshift-monitoring/node-exporter-t64f9" Jul 10 02:12:26.292346 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.292296 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fa530c6a-9f5d-4919-bb59-f3e7752a6c79-root\") pod \"node-exporter-t64f9\" (UID: \"fa530c6a-9f5d-4919-bb59-f3e7752a6c79\") " pod="openshift-monitoring/node-exporter-t64f9" Jul 10 02:12:26.292346 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.292320 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4642e40c-4306-467a-b96a-87311adc28e5-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-nznmv\" (UID: \"4642e40c-4306-467a-b96a-87311adc28e5\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nznmv" Jul 10 02:12:26.293003 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.292937 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef076029-9853-4835-b6ac-ee0a295c23fc-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-tplmw\" (UID: \"ef076029-9853-4835-b6ac-ee0a295c23fc\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tplmw" Jul 10 02:12:26.294072 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.293966 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4642e40c-4306-467a-b96a-87311adc28e5-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-nznmv\" (UID: \"4642e40c-4306-467a-b96a-87311adc28e5\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nznmv" Jul 10 02:12:26.294247 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.294224 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fa530c6a-9f5d-4919-bb59-f3e7752a6c79-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t64f9\" (UID: \"fa530c6a-9f5d-4919-bb59-f3e7752a6c79\") " pod="openshift-monitoring/node-exporter-t64f9" Jul 10 02:12:26.294970 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.294953 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef076029-9853-4835-b6ac-ee0a295c23fc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-tplmw\" (UID: \"ef076029-9853-4835-b6ac-ee0a295c23fc\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tplmw" Jul 10 02:12:26.295060 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.295016 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ef076029-9853-4835-b6ac-ee0a295c23fc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-tplmw\" (UID: \"ef076029-9853-4835-b6ac-ee0a295c23fc\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tplmw" Jul 10 02:12:26.295923 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.295901 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4642e40c-4306-467a-b96a-87311adc28e5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-nznmv\" (UID: \"4642e40c-4306-467a-b96a-87311adc28e5\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nznmv" Jul 10 02:12:26.301075 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.301053 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5d54\" (UniqueName: \"kubernetes.io/projected/4642e40c-4306-467a-b96a-87311adc28e5-kube-api-access-w5d54\") pod \"kube-state-metrics-785778ff74-nznmv\" (UID: \"4642e40c-4306-467a-b96a-87311adc28e5\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nznmv" Jul 10 02:12:26.301375 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.301358 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck2sx\" (UniqueName: \"kubernetes.io/projected/ef076029-9853-4835-b6ac-ee0a295c23fc-kube-api-access-ck2sx\") pod \"openshift-state-metrics-84bd68b896-tplmw\" (UID: \"ef076029-9853-4835-b6ac-ee0a295c23fc\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tplmw" Jul 10 02:12:26.301478 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.301395 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj529\" (UniqueName: \"kubernetes.io/projected/fa530c6a-9f5d-4919-bb59-f3e7752a6c79-kube-api-access-xj529\") pod \"node-exporter-t64f9\" (UID: \"fa530c6a-9f5d-4919-bb59-f3e7752a6c79\") " pod="openshift-monitoring/node-exporter-t64f9" Jul 10 02:12:26.320642 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.320618 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:26.401911 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.401817 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tplmw" Jul 10 02:12:26.419425 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.419336 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:26.449460 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.449408 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-nznmv" Jul 10 02:12:26.518898 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.518568 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:26.564542 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.564506 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-tplmw"] Jul 10 02:12:26.607267 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.607237 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-nznmv"] Jul 10 02:12:26.613691 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:12:26.613666 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4642e40c_4306_467a_b96a_87311adc28e5.slice/crio-4fc00e29b765a68060875bcca992e4d76bfcc4c33da699ced6f3cfe089d22716 WatchSource:0}: Error finding container 4fc00e29b765a68060875bcca992e4d76bfcc4c33da699ced6f3cfe089d22716: Status 404 returned error can't find the container with id 4fc00e29b765a68060875bcca992e4d76bfcc4c33da699ced6f3cfe089d22716 Jul 10 02:12:26.796358 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.796320 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fa530c6a-9f5d-4919-bb59-f3e7752a6c79-node-exporter-tls\") pod \"node-exporter-t64f9\" (UID: \"fa530c6a-9f5d-4919-bb59-f3e7752a6c79\") " pod="openshift-monitoring/node-exporter-t64f9" Jul 10 02:12:26.797644 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.797610 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tplmw" event={"ID":"ef076029-9853-4835-b6ac-ee0a295c23fc","Type":"ContainerStarted","Data":"b856753278cb49275db1208b668fe51015cade89955094a6e4c1225866d882ff"} Jul 10 02:12:26.797784 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.797712 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tplmw" event={"ID":"ef076029-9853-4835-b6ac-ee0a295c23fc","Type":"ContainerStarted","Data":"48c4107b67d51a7479983c45e30cd331d15ec4d35346c6fb2c2c06661947c613"} Jul 10 02:12:26.797784 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.797745 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tplmw" event={"ID":"ef076029-9853-4835-b6ac-ee0a295c23fc","Type":"ContainerStarted","Data":"b0364eaa619112b3d4711b1b1c94d99d4019af324d13dd17a1213d60112f7392"} Jul 10 02:12:26.798873 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.798845 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-nznmv" event={"ID":"4642e40c-4306-467a-b96a-87311adc28e5","Type":"ContainerStarted","Data":"4fc00e29b765a68060875bcca992e4d76bfcc4c33da699ced6f3cfe089d22716"} Jul 10 02:12:26.798995 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.798977 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fa530c6a-9f5d-4919-bb59-f3e7752a6c79-node-exporter-tls\") pod \"node-exporter-t64f9\" (UID: \"fa530c6a-9f5d-4919-bb59-f3e7752a6c79\") " pod="openshift-monitoring/node-exporter-t64f9" Jul 10 02:12:26.836886 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:26.836851 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:27.024904 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:27.024880 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-t64f9" Jul 10 02:12:27.034219 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:12:27.034192 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa530c6a_9f5d_4919_bb59_f3e7752a6c79.slice/crio-e7645132722349e514dd946f909dd1d0badfd18987656ac98ff890d84578241e WatchSource:0}: Error finding container e7645132722349e514dd946f909dd1d0badfd18987656ac98ff890d84578241e: Status 404 returned error can't find the container with id e7645132722349e514dd946f909dd1d0badfd18987656ac98ff890d84578241e Jul 10 02:12:27.804418 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:27.804377 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t64f9" event={"ID":"fa530c6a-9f5d-4919-bb59-f3e7752a6c79","Type":"ContainerStarted","Data":"e7645132722349e514dd946f909dd1d0badfd18987656ac98ff890d84578241e"} Jul 10 02:12:28.180831 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:28.180751 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:28.811065 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:28.811032 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tplmw" event={"ID":"ef076029-9853-4835-b6ac-ee0a295c23fc","Type":"ContainerStarted","Data":"77d6a4895e8f8d51b5405c8061ef759cec5056c26a25bd99018c274c0e44473c"} Jul 10 02:12:28.813143 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:28.813118 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-nznmv" event={"ID":"4642e40c-4306-467a-b96a-87311adc28e5","Type":"ContainerStarted","Data":"a724adb8fab05a4fb92083d56cf858d51147bd0ab265f200c6d5bd9eeb4ce6ef"} Jul 10 02:12:28.813274 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:28.813175 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-nznmv" event={"ID":"4642e40c-4306-467a-b96a-87311adc28e5","Type":"ContainerStarted","Data":"e32c074e3a73dd9d5b6b1f4b37d4880254be9b0f15121dd7406f51761d9a698b"} Jul 10 02:12:28.813274 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:28.813203 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-nznmv" event={"ID":"4642e40c-4306-467a-b96a-87311adc28e5","Type":"ContainerStarted","Data":"e8eb4e2f9cf78cdcf87ae319273590f2d529fb32117cf551609ba76d90279143"} Jul 10 02:12:28.814564 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:28.814542 2566 generic.go:356] "Generic (PLEG): container finished" podID="fa530c6a-9f5d-4919-bb59-f3e7752a6c79" containerID="228d211d9e042f770d0f7aa8a8b209e5c7d775dbf9a5f30e5d83f2e143485f21" exitCode=0 Jul 10 02:12:28.814642 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:28.814583 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t64f9" event={"ID":"fa530c6a-9f5d-4919-bb59-f3e7752a6c79","Type":"ContainerDied","Data":"228d211d9e042f770d0f7aa8a8b209e5c7d775dbf9a5f30e5d83f2e143485f21"} Jul 10 02:12:28.832387 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:28.832336 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-tplmw" podStartSLOduration=1.215416213 podStartE2EDuration="2.832318913s" totalImagesPullingTime="1.6169027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:12:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:12:28.830428207 +0000 UTC m=+280.406705297" watchObservedRunningTime="2026-07-10 02:12:28.832318913 +0000 UTC m=+280.408596001" Jul 10 02:12:28.870772 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:28.870721 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-nznmv" podStartSLOduration=1.180298552 podStartE2EDuration="2.870702871s" totalImagesPullingTime="1.690404319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:12:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:12:28.869404165 +0000 UTC m=+280.445681268" watchObservedRunningTime="2026-07-10 02:12:28.870702871 +0000 UTC m=+280.446979958" Jul 10 02:12:29.078843 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:29.078748 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:29.180012 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:29.179980 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:29.269549 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:29.269494 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:29.510029 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:29.509952 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:29.819409 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:29.819376 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t64f9" event={"ID":"fa530c6a-9f5d-4919-bb59-f3e7752a6c79","Type":"ContainerStarted","Data":"f344cc1ce4c396ed139b4ff4d0c9cc7de500048cd4eb1d3daf6919b5b6a08b26"} Jul 10 02:12:29.819558 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:29.819465 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t64f9" event={"ID":"fa530c6a-9f5d-4919-bb59-f3e7752a6c79","Type":"ContainerStarted","Data":"621317aa4e9a1d4b516f815c624604485693543dc39184522a02ce3b47ac1f34"} Jul 10 02:12:29.843426 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:29.843378 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-t64f9" podStartSLOduration=2.48442523 podStartE2EDuration="3.843364216s" totalImagesPullingTime="1.273100809s" totalInitContainerRuntime="85.838177ms" isStatefulPod=true podCreationTimestamp="2026-07-10 02:12:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:12:29.84248394 +0000 UTC m=+281.418761026" watchObservedRunningTime="2026-07-10 02:12:29.843364216 +0000 UTC m=+281.419641349" Jul 10 02:12:30.428291 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:30.428258 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7bd9d9f484-mmhsv"] Jul 10 02:12:30.430906 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:30.430889 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7bd9d9f484-mmhsv" Jul 10 02:12:30.435356 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:30.435329 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 10 02:12:30.435492 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:30.435354 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 10 02:12:30.435492 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:30.435354 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-jh4vw\"" Jul 10 02:12:30.435492 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:30.435388 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 10 02:12:30.435705 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:30.435388 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 10 02:12:30.435786 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:30.435772 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-c84nh8tk49jdq\"" Jul 10 02:12:30.440261 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:30.440241 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7bd9d9f484-mmhsv"] Jul 10 02:12:30.526868 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:30.526844 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7a42cfbb-60df-4742-80e4-c637ea5555fd-client-ca-bundle\") pod \"metrics-server-7bd9d9f484-mmhsv\" (UID: \"7a42cfbb-60df-4742-80e4-c637ea5555fd\") " pod="openshift-monitoring/metrics-server-7bd9d9f484-mmhsv" Jul 10 02:12:30.527000 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:30.526888 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a42cfbb-60df-4742-80e4-c637ea5555fd-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7bd9d9f484-mmhsv\" (UID: \"7a42cfbb-60df-4742-80e4-c637ea5555fd\") " pod="openshift-monitoring/metrics-server-7bd9d9f484-mmhsv" Jul 10 02:12:30.527000 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:30.526911 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7a42cfbb-60df-4742-80e4-c637ea5555fd-audit-log\") pod \"metrics-server-7bd9d9f484-mmhsv\" (UID: \"7a42cfbb-60df-4742-80e4-c637ea5555fd\") " pod="openshift-monitoring/metrics-server-7bd9d9f484-mmhsv" Jul 10 02:12:30.527094 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:30.526990 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7a42cfbb-60df-4742-80e4-c637ea5555fd-secret-metrics-server-client-certs\") pod \"metrics-server-7bd9d9f484-mmhsv\" (UID: \"7a42cfbb-60df-4742-80e4-c637ea5555fd\") " pod="openshift-monitoring/metrics-server-7bd9d9f484-mmhsv" Jul 10 02:12:30.527144 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:30.527092 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7a42cfbb-60df-4742-80e4-c637ea5555fd-metrics-server-audit-profiles\") pod \"metrics-server-7bd9d9f484-mmhsv\" (UID: \"7a42cfbb-60df-4742-80e4-c637ea5555fd\") " pod="openshift-monitoring/metrics-server-7bd9d9f484-mmhsv" Jul 10 02:12:30.527144 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:30.527110 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfxx4\" (UniqueName: \"kubernetes.io/projected/7a42cfbb-60df-4742-80e4-c637ea5555fd-kube-api-access-sfxx4\") pod \"metrics-server-7bd9d9f484-mmhsv\" (UID: \"7a42cfbb-60df-4742-80e4-c637ea5555fd\") " pod="openshift-monitoring/metrics-server-7bd9d9f484-mmhsv" Jul 10 02:12:30.527144 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:30.527126 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7a42cfbb-60df-4742-80e4-c637ea5555fd-secret-metrics-server-tls\") pod \"metrics-server-7bd9d9f484-mmhsv\" (UID: \"7a42cfbb-60df-4742-80e4-c637ea5555fd\") " pod="openshift-monitoring/metrics-server-7bd9d9f484-mmhsv" Jul 10 02:12:30.592536 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:30.592500 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:30.628351 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:30.628326 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a42cfbb-60df-4742-80e4-c637ea5555fd-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7bd9d9f484-mmhsv\" (UID: \"7a42cfbb-60df-4742-80e4-c637ea5555fd\") " pod="openshift-monitoring/metrics-server-7bd9d9f484-mmhsv" Jul 10 02:12:30.628475 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:30.628355 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7a42cfbb-60df-4742-80e4-c637ea5555fd-audit-log\") pod \"metrics-server-7bd9d9f484-mmhsv\" (UID: \"7a42cfbb-60df-4742-80e4-c637ea5555fd\") " pod="openshift-monitoring/metrics-server-7bd9d9f484-mmhsv" Jul 10 02:12:30.628475 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:30.628376 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7a42cfbb-60df-4742-80e4-c637ea5555fd-secret-metrics-server-client-certs\") pod \"metrics-server-7bd9d9f484-mmhsv\" (UID: \"7a42cfbb-60df-4742-80e4-c637ea5555fd\") " pod="openshift-monitoring/metrics-server-7bd9d9f484-mmhsv" Jul 10 02:12:30.628565 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:30.628502 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7a42cfbb-60df-4742-80e4-c637ea5555fd-metrics-server-audit-profiles\") pod \"metrics-server-7bd9d9f484-mmhsv\" (UID: \"7a42cfbb-60df-4742-80e4-c637ea5555fd\") " pod="openshift-monitoring/metrics-server-7bd9d9f484-mmhsv" Jul 10 02:12:30.628565 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:30.628531 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfxx4\" (UniqueName: \"kubernetes.io/projected/7a42cfbb-60df-4742-80e4-c637ea5555fd-kube-api-access-sfxx4\") pod \"metrics-server-7bd9d9f484-mmhsv\" (UID: \"7a42cfbb-60df-4742-80e4-c637ea5555fd\") " pod="openshift-monitoring/metrics-server-7bd9d9f484-mmhsv" Jul 10 02:12:30.628565 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:30.628560 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7a42cfbb-60df-4742-80e4-c637ea5555fd-secret-metrics-server-tls\") pod \"metrics-server-7bd9d9f484-mmhsv\" (UID: \"7a42cfbb-60df-4742-80e4-c637ea5555fd\") " pod="openshift-monitoring/metrics-server-7bd9d9f484-mmhsv" Jul 10 02:12:30.628704 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:30.628684 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7a42cfbb-60df-4742-80e4-c637ea5555fd-client-ca-bundle\") pod \"metrics-server-7bd9d9f484-mmhsv\" (UID: \"7a42cfbb-60df-4742-80e4-c637ea5555fd\") " pod="openshift-monitoring/metrics-server-7bd9d9f484-mmhsv" Jul 10 02:12:30.628800 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:30.628777 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7a42cfbb-60df-4742-80e4-c637ea5555fd-audit-log\") pod \"metrics-server-7bd9d9f484-mmhsv\" (UID: \"7a42cfbb-60df-4742-80e4-c637ea5555fd\") " pod="openshift-monitoring/metrics-server-7bd9d9f484-mmhsv" Jul 10 02:12:30.629170 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:30.629145 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a42cfbb-60df-4742-80e4-c637ea5555fd-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7bd9d9f484-mmhsv\" (UID: \"7a42cfbb-60df-4742-80e4-c637ea5555fd\") " pod="openshift-monitoring/metrics-server-7bd9d9f484-mmhsv" Jul 10 02:12:30.629441 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:30.629420 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7a42cfbb-60df-4742-80e4-c637ea5555fd-metrics-server-audit-profiles\") pod \"metrics-server-7bd9d9f484-mmhsv\" (UID: \"7a42cfbb-60df-4742-80e4-c637ea5555fd\") " pod="openshift-monitoring/metrics-server-7bd9d9f484-mmhsv" Jul 10 02:12:30.630900 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:30.630873 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7a42cfbb-60df-4742-80e4-c637ea5555fd-secret-metrics-server-tls\") pod \"metrics-server-7bd9d9f484-mmhsv\" (UID: \"7a42cfbb-60df-4742-80e4-c637ea5555fd\") " pod="openshift-monitoring/metrics-server-7bd9d9f484-mmhsv" Jul 10 02:12:30.630987 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:30.630957 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7a42cfbb-60df-4742-80e4-c637ea5555fd-client-ca-bundle\") pod \"metrics-server-7bd9d9f484-mmhsv\" (UID: \"7a42cfbb-60df-4742-80e4-c637ea5555fd\") " pod="openshift-monitoring/metrics-server-7bd9d9f484-mmhsv" Jul 10 02:12:30.631037 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:30.631020 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7a42cfbb-60df-4742-80e4-c637ea5555fd-secret-metrics-server-client-certs\") pod \"metrics-server-7bd9d9f484-mmhsv\" (UID: \"7a42cfbb-60df-4742-80e4-c637ea5555fd\") " pod="openshift-monitoring/metrics-server-7bd9d9f484-mmhsv" Jul 10 02:12:30.640574 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:30.640554 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfxx4\" (UniqueName: \"kubernetes.io/projected/7a42cfbb-60df-4742-80e4-c637ea5555fd-kube-api-access-sfxx4\") pod \"metrics-server-7bd9d9f484-mmhsv\" (UID: \"7a42cfbb-60df-4742-80e4-c637ea5555fd\") " pod="openshift-monitoring/metrics-server-7bd9d9f484-mmhsv" Jul 10 02:12:30.693008 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:30.692938 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:30.740216 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:30.740194 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7bd9d9f484-mmhsv" Jul 10 02:12:30.746740 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:30.746708 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:30.843524 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:30.843447 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:30.852252 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:30.852224 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-5vbk8"] Jul 10 02:12:30.855709 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:30.854949 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-5vbk8" Jul 10 02:12:30.858033 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:30.858016 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 10 02:12:30.858113 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:30.858061 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-f8b8k\"" Jul 10 02:12:30.864205 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:30.864182 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-5vbk8"] Jul 10 02:12:30.875734 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:30.875714 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7bd9d9f484-mmhsv"] Jul 10 02:12:30.879908 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:12:30.879882 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a42cfbb_60df_4742_80e4_c637ea5555fd.slice/crio-81b23667eea106b87a8771761c880f81811cd649eb8ce29a1522f16080ce68ac WatchSource:0}: Error finding container 81b23667eea106b87a8771761c880f81811cd649eb8ce29a1522f16080ce68ac: Status 404 returned error can't find the container with id 81b23667eea106b87a8771761c880f81811cd649eb8ce29a1522f16080ce68ac Jul 10 02:12:30.931713 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:30.931688 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4q2t\" (UniqueName: \"kubernetes.io/projected/ac1608c3-32c1-4ccf-9c09-300a8524a4ae-kube-api-access-p4q2t\") pod \"monitoring-plugin-5f68df7f58-5vbk8\" (UID: \"ac1608c3-32c1-4ccf-9c09-300a8524a4ae\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-5vbk8" Jul 10 02:12:30.931849 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:30.931826 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ac1608c3-32c1-4ccf-9c09-300a8524a4ae-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-5vbk8\" (UID: \"ac1608c3-32c1-4ccf-9c09-300a8524a4ae\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-5vbk8" Jul 10 02:12:31.032889 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:31.032867 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4q2t\" (UniqueName: \"kubernetes.io/projected/ac1608c3-32c1-4ccf-9c09-300a8524a4ae-kube-api-access-p4q2t\") pod \"monitoring-plugin-5f68df7f58-5vbk8\" (UID: \"ac1608c3-32c1-4ccf-9c09-300a8524a4ae\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-5vbk8" Jul 10 02:12:31.032989 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:31.032906 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ac1608c3-32c1-4ccf-9c09-300a8524a4ae-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-5vbk8\" (UID: \"ac1608c3-32c1-4ccf-9c09-300a8524a4ae\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-5vbk8" Jul 10 02:12:31.033029 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:12:31.032995 2566 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 10 02:12:31.033063 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:12:31.033039 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac1608c3-32c1-4ccf-9c09-300a8524a4ae-monitoring-plugin-cert podName:ac1608c3-32c1-4ccf-9c09-300a8524a4ae nodeName:}" failed. No retries permitted until 2026-07-10 02:12:31.53302515 +0000 UTC m=+283.109302234 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/ac1608c3-32c1-4ccf-9c09-300a8524a4ae-monitoring-plugin-cert") pod "monitoring-plugin-5f68df7f58-5vbk8" (UID: "ac1608c3-32c1-4ccf-9c09-300a8524a4ae") : secret "monitoring-plugin-cert" not found Jul 10 02:12:31.042448 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:31.042428 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4q2t\" (UniqueName: \"kubernetes.io/projected/ac1608c3-32c1-4ccf-9c09-300a8524a4ae-kube-api-access-p4q2t\") pod \"monitoring-plugin-5f68df7f58-5vbk8\" (UID: \"ac1608c3-32c1-4ccf-9c09-300a8524a4ae\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-5vbk8" Jul 10 02:12:31.228295 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:31.228270 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:31.536776 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:31.536737 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ac1608c3-32c1-4ccf-9c09-300a8524a4ae-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-5vbk8\" (UID: \"ac1608c3-32c1-4ccf-9c09-300a8524a4ae\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-5vbk8" Jul 10 02:12:31.539058 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:31.539038 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ac1608c3-32c1-4ccf-9c09-300a8524a4ae-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-5vbk8\" (UID: \"ac1608c3-32c1-4ccf-9c09-300a8524a4ae\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-5vbk8" Jul 10 02:12:31.765917 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:31.765880 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-5vbk8" Jul 10 02:12:31.829406 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:31.829377 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7bd9d9f484-mmhsv" event={"ID":"7a42cfbb-60df-4742-80e4-c637ea5555fd","Type":"ContainerStarted","Data":"81b23667eea106b87a8771761c880f81811cd649eb8ce29a1522f16080ce68ac"} Jul 10 02:12:31.891919 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:31.891888 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-5vbk8"] Jul 10 02:12:31.896430 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:12:31.896398 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac1608c3_32c1_4ccf_9c09_300a8524a4ae.slice/crio-2abdca54649458b8dbb92d18c1b5c593c44e8490521c3af9d0d8e638218b29fa WatchSource:0}: Error finding container 2abdca54649458b8dbb92d18c1b5c593c44e8490521c3af9d0d8e638218b29fa: Status 404 returned error can't find the container with id 2abdca54649458b8dbb92d18c1b5c593c44e8490521c3af9d0d8e638218b29fa Jul 10 02:12:32.150559 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:32.150470 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:32.252566 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:32.252529 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:32.406120 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:32.406029 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:32.839277 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:32.839239 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-5vbk8" event={"ID":"ac1608c3-32c1-4ccf-9c09-300a8524a4ae","Type":"ContainerStarted","Data":"2abdca54649458b8dbb92d18c1b5c593c44e8490521c3af9d0d8e638218b29fa"} Jul 10 02:12:32.840667 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:32.840638 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7bd9d9f484-mmhsv" event={"ID":"7a42cfbb-60df-4742-80e4-c637ea5555fd","Type":"ContainerStarted","Data":"0cdb5bd0c1d4dbebfc2bd96b99397cea9448fa20444839dca78a1bbcb3595d35"} Jul 10 02:12:32.861087 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:32.861042 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7bd9d9f484-mmhsv" podStartSLOduration=1.106082174 podStartE2EDuration="2.86103038s" totalImagesPullingTime="1.754948206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:12:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:12:32.859855735 +0000 UTC m=+284.436132821" watchObservedRunningTime="2026-07-10 02:12:32.86103038 +0000 UTC m=+284.437307465" Jul 10 02:12:33.465877 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:12:33.465841 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-pl7t5" podUID="9ce933c0-a5ae-4f64-83af-ec2eef8371b0" Jul 10 02:12:33.843515 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:33.843491 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pl7t5" Jul 10 02:12:34.578022 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:34.577982 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:34.675272 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:34.675239 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:34.848453 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:34.848374 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-5vbk8" event={"ID":"ac1608c3-32c1-4ccf-9c09-300a8524a4ae","Type":"ContainerStarted","Data":"3bcd6d5f6fc537a46dd7c29a20e5d751bca865df0150c9c923f60df2ac1ab333"} Jul 10 02:12:34.848836 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:34.848529 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-5vbk8" Jul 10 02:12:34.853140 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:34.853116 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-5vbk8" Jul 10 02:12:34.867099 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:34.867064 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-5vbk8" podStartSLOduration=2.953696687 podStartE2EDuration="4.867051464s" totalImagesPullingTime="1.913354777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:12:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:12:34.865052071 +0000 UTC m=+286.441329157" watchObservedRunningTime="2026-07-10 02:12:34.867051464 +0000 UTC m=+286.443328910" Jul 10 02:12:36.355923 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:36.355890 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:36.886963 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:36.886936 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ce933c0-a5ae-4f64-83af-ec2eef8371b0-metrics-tls\") pod \"dns-default-pl7t5\" (UID: \"9ce933c0-a5ae-4f64-83af-ec2eef8371b0\") " pod="openshift-dns/dns-default-pl7t5" Jul 10 02:12:36.889168 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:36.889144 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ce933c0-a5ae-4f64-83af-ec2eef8371b0-metrics-tls\") pod \"dns-default-pl7t5\" (UID: \"9ce933c0-a5ae-4f64-83af-ec2eef8371b0\") " pod="openshift-dns/dns-default-pl7t5" Jul 10 02:12:37.147140 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:37.147079 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-g8kf5\"" Jul 10 02:12:37.155416 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:37.155400 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pl7t5" Jul 10 02:12:37.281479 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:37.281344 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pl7t5"] Jul 10 02:12:37.283954 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:12:37.283928 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ce933c0_a5ae_4f64_83af_ec2eef8371b0.slice/crio-ce169c926e11091c6d30a2b8d4bc8c6f256b97cf09275df135413b8d90c4306a WatchSource:0}: Error finding container ce169c926e11091c6d30a2b8d4bc8c6f256b97cf09275df135413b8d90c4306a: Status 404 returned error can't find the container with id ce169c926e11091c6d30a2b8d4bc8c6f256b97cf09275df135413b8d90c4306a Jul 10 02:12:37.860121 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:37.860086 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pl7t5" event={"ID":"9ce933c0-a5ae-4f64-83af-ec2eef8371b0","Type":"ContainerStarted","Data":"ce169c926e11091c6d30a2b8d4bc8c6f256b97cf09275df135413b8d90c4306a"} Jul 10 02:12:40.872489 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:40.872454 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pl7t5" event={"ID":"9ce933c0-a5ae-4f64-83af-ec2eef8371b0","Type":"ContainerStarted","Data":"ccfeaee30e3f57da79fc7719cf8b982295dad69d595d8c8b22e83f751996601c"} Jul 10 02:12:40.872861 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:40.872506 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pl7t5" event={"ID":"9ce933c0-a5ae-4f64-83af-ec2eef8371b0","Type":"ContainerStarted","Data":"562eb29765c2927e604cec194ce0e2b8961b23cc59d4212399862cefde53edc8"} Jul 10 02:12:40.872861 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:40.872589 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pl7t5" Jul 10 02:12:40.892602 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:40.892564 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-pl7t5" podStartSLOduration=252.439110054 podStartE2EDuration="4m14.892552066s" totalImagesPullingTime="2.453442012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:08:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:12:40.891030199 +0000 UTC m=+292.467307287" watchObservedRunningTime="2026-07-10 02:12:40.892552066 +0000 UTC m=+292.468829142" Jul 10 02:12:41.133759 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:41.133689 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-r7zjf"] Jul 10 02:12:41.136448 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:41.136430 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-r7zjf" Jul 10 02:12:41.137370 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:41.137340 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:41.139094 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:41.139077 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 10 02:12:41.139186 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:41.139078 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 10 02:12:41.139238 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:41.139185 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-bkdrd\"" Jul 10 02:12:41.145984 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:41.145966 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-r7zjf"] Jul 10 02:12:41.199560 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:41.199535 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:41.221833 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:41.221814 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld5vv\" (UniqueName: \"kubernetes.io/projected/86b9e286-1e9b-43dd-9ff4-ca61dce4bc6c-kube-api-access-ld5vv\") pod \"downloads-68c9cbb497-r7zjf\" (UID: \"86b9e286-1e9b-43dd-9ff4-ca61dce4bc6c\") " pod="openshift-console/downloads-68c9cbb497-r7zjf" Jul 10 02:12:41.322431 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:41.322410 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld5vv\" (UniqueName: \"kubernetes.io/projected/86b9e286-1e9b-43dd-9ff4-ca61dce4bc6c-kube-api-access-ld5vv\") pod \"downloads-68c9cbb497-r7zjf\" (UID: \"86b9e286-1e9b-43dd-9ff4-ca61dce4bc6c\") " pod="openshift-console/downloads-68c9cbb497-r7zjf" Jul 10 02:12:41.332372 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:41.332346 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld5vv\" (UniqueName: \"kubernetes.io/projected/86b9e286-1e9b-43dd-9ff4-ca61dce4bc6c-kube-api-access-ld5vv\") pod \"downloads-68c9cbb497-r7zjf\" (UID: \"86b9e286-1e9b-43dd-9ff4-ca61dce4bc6c\") " pod="openshift-console/downloads-68c9cbb497-r7zjf" Jul 10 02:12:41.387832 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:41.387768 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:41.445700 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:41.445680 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-r7zjf" Jul 10 02:12:41.575243 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:41.575174 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-r7zjf"] Jul 10 02:12:41.577290 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:12:41.577260 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86b9e286_1e9b_43dd_9ff4_ca61dce4bc6c.slice/crio-d67ee1efe13203836871205ab2ca5e7429e36ad96603a374c693e845289f5ce1 WatchSource:0}: Error finding container d67ee1efe13203836871205ab2ca5e7429e36ad96603a374c693e845289f5ce1: Status 404 returned error can't find the container with id d67ee1efe13203836871205ab2ca5e7429e36ad96603a374c693e845289f5ce1 Jul 10 02:12:41.765358 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:41.765275 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:41.822876 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:41.822842 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:41.876718 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:41.876692 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-r7zjf" event={"ID":"86b9e286-1e9b-43dd-9ff4-ca61dce4bc6c","Type":"ContainerStarted","Data":"d67ee1efe13203836871205ab2ca5e7429e36ad96603a374c693e845289f5ce1"} Jul 10 02:12:42.174854 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:42.174817 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:42.581930 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:42.581895 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:42.786964 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:42.786922 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:42.883661 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:42.883559 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:42.939118 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:42.939091 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:45.407031 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:45.406998 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:45.481092 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:45.481059 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:46.005870 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:12:46.005827 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-6hfgs" podUID="8bf086af-02b9-43f2-935f-474133fef4cf" Jul 10 02:12:46.560229 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:46.560195 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:46.617567 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:46.617526 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:48.433041 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:48.433006 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d7b8c6c9d-h7w9q"] Jul 10 02:12:48.442679 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:48.442650 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d7b8c6c9d-h7w9q" Jul 10 02:12:48.446532 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:48.446501 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d7b8c6c9d-h7w9q"] Jul 10 02:12:48.446708 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:48.446690 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 10 02:12:48.447379 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:48.447354 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 10 02:12:48.448751 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:48.448727 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-4rqnq\"" Jul 10 02:12:48.451517 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:48.450237 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 10 02:12:48.451517 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:48.450345 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 10 02:12:48.451517 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:48.451189 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 10 02:12:48.585235 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:48.585204 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5361f125-2d98-409a-8966-f54d6f90066b-console-config\") pod \"console-5d7b8c6c9d-h7w9q\" (UID: \"5361f125-2d98-409a-8966-f54d6f90066b\") " pod="openshift-console/console-5d7b8c6c9d-h7w9q" Jul 10 02:12:48.585366 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:48.585242 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5361f125-2d98-409a-8966-f54d6f90066b-service-ca\") pod \"console-5d7b8c6c9d-h7w9q\" (UID: \"5361f125-2d98-409a-8966-f54d6f90066b\") " pod="openshift-console/console-5d7b8c6c9d-h7w9q" Jul 10 02:12:48.585366 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:48.585277 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5361f125-2d98-409a-8966-f54d6f90066b-console-serving-cert\") pod \"console-5d7b8c6c9d-h7w9q\" (UID: \"5361f125-2d98-409a-8966-f54d6f90066b\") " pod="openshift-console/console-5d7b8c6c9d-h7w9q" Jul 10 02:12:48.585366 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:48.585339 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5361f125-2d98-409a-8966-f54d6f90066b-oauth-serving-cert\") pod \"console-5d7b8c6c9d-h7w9q\" (UID: \"5361f125-2d98-409a-8966-f54d6f90066b\") " pod="openshift-console/console-5d7b8c6c9d-h7w9q" Jul 10 02:12:48.585366 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:48.585359 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5361f125-2d98-409a-8966-f54d6f90066b-console-oauth-config\") pod \"console-5d7b8c6c9d-h7w9q\" (UID: \"5361f125-2d98-409a-8966-f54d6f90066b\") " pod="openshift-console/console-5d7b8c6c9d-h7w9q" Jul 10 02:12:48.585496 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:48.585428 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46hq2\" (UniqueName: \"kubernetes.io/projected/5361f125-2d98-409a-8966-f54d6f90066b-kube-api-access-46hq2\") pod \"console-5d7b8c6c9d-h7w9q\" (UID: \"5361f125-2d98-409a-8966-f54d6f90066b\") " pod="openshift-console/console-5d7b8c6c9d-h7w9q" Jul 10 02:12:48.685885 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:48.685803 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5361f125-2d98-409a-8966-f54d6f90066b-console-config\") pod \"console-5d7b8c6c9d-h7w9q\" (UID: \"5361f125-2d98-409a-8966-f54d6f90066b\") " pod="openshift-console/console-5d7b8c6c9d-h7w9q" Jul 10 02:12:48.685885 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:48.685841 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5361f125-2d98-409a-8966-f54d6f90066b-service-ca\") pod \"console-5d7b8c6c9d-h7w9q\" (UID: \"5361f125-2d98-409a-8966-f54d6f90066b\") " pod="openshift-console/console-5d7b8c6c9d-h7w9q" Jul 10 02:12:48.686086 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:48.685889 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5361f125-2d98-409a-8966-f54d6f90066b-console-serving-cert\") pod \"console-5d7b8c6c9d-h7w9q\" (UID: \"5361f125-2d98-409a-8966-f54d6f90066b\") " pod="openshift-console/console-5d7b8c6c9d-h7w9q" Jul 10 02:12:48.686086 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:48.685941 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5361f125-2d98-409a-8966-f54d6f90066b-oauth-serving-cert\") pod \"console-5d7b8c6c9d-h7w9q\" (UID: \"5361f125-2d98-409a-8966-f54d6f90066b\") " pod="openshift-console/console-5d7b8c6c9d-h7w9q" Jul 10 02:12:48.686086 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:48.685970 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5361f125-2d98-409a-8966-f54d6f90066b-console-oauth-config\") pod \"console-5d7b8c6c9d-h7w9q\" (UID: \"5361f125-2d98-409a-8966-f54d6f90066b\") " pod="openshift-console/console-5d7b8c6c9d-h7w9q" Jul 10 02:12:48.686086 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:48.686010 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46hq2\" (UniqueName: \"kubernetes.io/projected/5361f125-2d98-409a-8966-f54d6f90066b-kube-api-access-46hq2\") pod \"console-5d7b8c6c9d-h7w9q\" (UID: \"5361f125-2d98-409a-8966-f54d6f90066b\") " pod="openshift-console/console-5d7b8c6c9d-h7w9q" Jul 10 02:12:48.686690 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:48.686656 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5361f125-2d98-409a-8966-f54d6f90066b-console-config\") pod \"console-5d7b8c6c9d-h7w9q\" (UID: \"5361f125-2d98-409a-8966-f54d6f90066b\") " pod="openshift-console/console-5d7b8c6c9d-h7w9q" Jul 10 02:12:48.686814 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:48.686719 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5361f125-2d98-409a-8966-f54d6f90066b-oauth-serving-cert\") pod \"console-5d7b8c6c9d-h7w9q\" (UID: \"5361f125-2d98-409a-8966-f54d6f90066b\") " pod="openshift-console/console-5d7b8c6c9d-h7w9q" Jul 10 02:12:48.686814 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:48.686770 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5361f125-2d98-409a-8966-f54d6f90066b-service-ca\") pod \"console-5d7b8c6c9d-h7w9q\" (UID: \"5361f125-2d98-409a-8966-f54d6f90066b\") " pod="openshift-console/console-5d7b8c6c9d-h7w9q" Jul 10 02:12:48.688722 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:48.688699 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5361f125-2d98-409a-8966-f54d6f90066b-console-serving-cert\") pod \"console-5d7b8c6c9d-h7w9q\" (UID: \"5361f125-2d98-409a-8966-f54d6f90066b\") " pod="openshift-console/console-5d7b8c6c9d-h7w9q" Jul 10 02:12:48.688848 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:48.688786 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5361f125-2d98-409a-8966-f54d6f90066b-console-oauth-config\") pod \"console-5d7b8c6c9d-h7w9q\" (UID: \"5361f125-2d98-409a-8966-f54d6f90066b\") " pod="openshift-console/console-5d7b8c6c9d-h7w9q" Jul 10 02:12:48.696013 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:48.695984 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46hq2\" (UniqueName: \"kubernetes.io/projected/5361f125-2d98-409a-8966-f54d6f90066b-kube-api-access-46hq2\") pod \"console-5d7b8c6c9d-h7w9q\" (UID: \"5361f125-2d98-409a-8966-f54d6f90066b\") " pod="openshift-console/console-5d7b8c6c9d-h7w9q" Jul 10 02:12:48.757323 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:48.757291 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d7b8c6c9d-h7w9q" Jul 10 02:12:48.846058 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:48.846022 2566 kubelet.go:1648] "Image garbage collection succeeded" Jul 10 02:12:48.915143 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:48.915122 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d7b8c6c9d-h7w9q"] Jul 10 02:12:48.917767 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:12:48.917731 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5361f125_2d98_409a_8966_f54d6f90066b.slice/crio-ae625014a7ac1ca6be3c7c0e38d5b4fcc8d57a31d829723e23eabbbf9c1d7594 WatchSource:0}: Error finding container ae625014a7ac1ca6be3c7c0e38d5b4fcc8d57a31d829723e23eabbbf9c1d7594: Status 404 returned error can't find the container with id ae625014a7ac1ca6be3c7c0e38d5b4fcc8d57a31d829723e23eabbbf9c1d7594 Jul 10 02:12:49.911708 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:49.911668 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d7b8c6c9d-h7w9q" event={"ID":"5361f125-2d98-409a-8966-f54d6f90066b","Type":"ContainerStarted","Data":"ae625014a7ac1ca6be3c7c0e38d5b4fcc8d57a31d829723e23eabbbf9c1d7594"} Jul 10 02:12:50.741207 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:50.741165 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7bd9d9f484-mmhsv" Jul 10 02:12:50.741413 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:50.741241 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7bd9d9f484-mmhsv" Jul 10 02:12:50.879123 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:50.879094 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pl7t5" Jul 10 02:12:53.242933 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:53.242902 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:53.341536 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:53.341487 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:55.639182 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:55.639109 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:55.740341 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:55.740303 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:59.807644 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:59.807570 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:59.905030 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:59.904989 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:12:59.955433 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:59.955402 2566 generic.go:356] "Generic (PLEG): container finished" podID="54ca438c-c22e-45df-8aa2-f653453022b0" containerID="8f05077f93ef57eddd26015b88403255b1e064050a37f3f4924c85d001389de7" exitCode=0 Jul 10 02:12:59.955634 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:59.955492 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-whpts" event={"ID":"54ca438c-c22e-45df-8aa2-f653453022b0","Type":"ContainerDied","Data":"8f05077f93ef57eddd26015b88403255b1e064050a37f3f4924c85d001389de7"} Jul 10 02:12:59.955944 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:59.955922 2566 scope.go:117] "RemoveContainer" containerID="8f05077f93ef57eddd26015b88403255b1e064050a37f3f4924c85d001389de7" Jul 10 02:12:59.957800 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:59.957592 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-r7zjf" event={"ID":"86b9e286-1e9b-43dd-9ff4-ca61dce4bc6c","Type":"ContainerStarted","Data":"33a8b19ecc9d6486faab781b4836a73e94c8f9a802077c52c1809cb7164f48d6"} Jul 10 02:12:59.957917 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:59.957858 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-r7zjf" Jul 10 02:12:59.959695 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:59.959668 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d7b8c6c9d-h7w9q" event={"ID":"5361f125-2d98-409a-8966-f54d6f90066b","Type":"ContainerStarted","Data":"189377c7de2d87ba37b378f67af6937179340d0268d295fd11bd6098a817a395"} Jul 10 02:12:59.964811 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:59.964790 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-r7zjf" Jul 10 02:12:59.996785 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:12:59.996730 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-r7zjf" podStartSLOduration=1.047622103 podStartE2EDuration="18.996710677s" totalImagesPullingTime="17.949088574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:12:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:12:59.995027301 +0000 UTC m=+311.571304388" watchObservedRunningTime="2026-07-10 02:12:59.996710677 +0000 UTC m=+311.572987766" Jul 10 02:13:00.006719 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:00.006691 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5fbbd6f97f-zm5fz"] Jul 10 02:13:00.015779 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:00.014701 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fbbd6f97f-zm5fz" Jul 10 02:13:00.015779 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:00.014783 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hfgs" Jul 10 02:13:00.020311 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:00.019707 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fbbd6f97f-zm5fz"] Jul 10 02:13:00.024120 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:00.024029 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 10 02:13:00.030432 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:00.030376 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5d7b8c6c9d-h7w9q" podStartSLOduration=1.468332922 podStartE2EDuration="12.030355567s" totalImagesPullingTime="10.562022645s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:12:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:13:00.030311122 +0000 UTC m=+311.606588209" watchObservedRunningTime="2026-07-10 02:13:00.030355567 +0000 UTC m=+311.606632654" Jul 10 02:13:00.039623 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:00.039480 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:13:00.098872 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:00.098588 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:13:00.190768 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:00.190738 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bd627785-40f4-4336-b29e-691de0575e7b-console-serving-cert\") pod \"console-5fbbd6f97f-zm5fz\" (UID: \"bd627785-40f4-4336-b29e-691de0575e7b\") " pod="openshift-console/console-5fbbd6f97f-zm5fz" Jul 10 02:13:00.190950 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:00.190787 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd627785-40f4-4336-b29e-691de0575e7b-trusted-ca-bundle\") pod \"console-5fbbd6f97f-zm5fz\" (UID: \"bd627785-40f4-4336-b29e-691de0575e7b\") " pod="openshift-console/console-5fbbd6f97f-zm5fz" Jul 10 02:13:00.190950 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:00.190866 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bd627785-40f4-4336-b29e-691de0575e7b-oauth-serving-cert\") pod \"console-5fbbd6f97f-zm5fz\" (UID: \"bd627785-40f4-4336-b29e-691de0575e7b\") " pod="openshift-console/console-5fbbd6f97f-zm5fz" Jul 10 02:13:00.190950 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:00.190944 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25hw8\" (UniqueName: \"kubernetes.io/projected/bd627785-40f4-4336-b29e-691de0575e7b-kube-api-access-25hw8\") pod \"console-5fbbd6f97f-zm5fz\" (UID: \"bd627785-40f4-4336-b29e-691de0575e7b\") " pod="openshift-console/console-5fbbd6f97f-zm5fz" Jul 10 02:13:00.191116 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:00.190996 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bd627785-40f4-4336-b29e-691de0575e7b-service-ca\") pod \"console-5fbbd6f97f-zm5fz\" (UID: \"bd627785-40f4-4336-b29e-691de0575e7b\") " pod="openshift-console/console-5fbbd6f97f-zm5fz" Jul 10 02:13:00.191116 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:00.191043 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bd627785-40f4-4336-b29e-691de0575e7b-console-config\") pod \"console-5fbbd6f97f-zm5fz\" (UID: \"bd627785-40f4-4336-b29e-691de0575e7b\") " pod="openshift-console/console-5fbbd6f97f-zm5fz" Jul 10 02:13:00.191116 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:00.191093 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bd627785-40f4-4336-b29e-691de0575e7b-console-oauth-config\") pod \"console-5fbbd6f97f-zm5fz\" (UID: \"bd627785-40f4-4336-b29e-691de0575e7b\") " pod="openshift-console/console-5fbbd6f97f-zm5fz" Jul 10 02:13:00.292574 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:00.292538 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bd627785-40f4-4336-b29e-691de0575e7b-console-config\") pod \"console-5fbbd6f97f-zm5fz\" (UID: \"bd627785-40f4-4336-b29e-691de0575e7b\") " pod="openshift-console/console-5fbbd6f97f-zm5fz" Jul 10 02:13:00.292799 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:00.292625 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bd627785-40f4-4336-b29e-691de0575e7b-console-oauth-config\") pod \"console-5fbbd6f97f-zm5fz\" (UID: \"bd627785-40f4-4336-b29e-691de0575e7b\") " pod="openshift-console/console-5fbbd6f97f-zm5fz" Jul 10 02:13:00.292799 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:00.292681 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bd627785-40f4-4336-b29e-691de0575e7b-console-serving-cert\") pod \"console-5fbbd6f97f-zm5fz\" (UID: \"bd627785-40f4-4336-b29e-691de0575e7b\") " pod="openshift-console/console-5fbbd6f97f-zm5fz" Jul 10 02:13:00.292799 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:00.292714 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd627785-40f4-4336-b29e-691de0575e7b-trusted-ca-bundle\") pod \"console-5fbbd6f97f-zm5fz\" (UID: \"bd627785-40f4-4336-b29e-691de0575e7b\") " pod="openshift-console/console-5fbbd6f97f-zm5fz" Jul 10 02:13:00.292799 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:00.292736 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bd627785-40f4-4336-b29e-691de0575e7b-oauth-serving-cert\") pod \"console-5fbbd6f97f-zm5fz\" (UID: \"bd627785-40f4-4336-b29e-691de0575e7b\") " pod="openshift-console/console-5fbbd6f97f-zm5fz" Jul 10 02:13:00.292799 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:00.292781 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25hw8\" (UniqueName: \"kubernetes.io/projected/bd627785-40f4-4336-b29e-691de0575e7b-kube-api-access-25hw8\") pod \"console-5fbbd6f97f-zm5fz\" (UID: \"bd627785-40f4-4336-b29e-691de0575e7b\") " pod="openshift-console/console-5fbbd6f97f-zm5fz" Jul 10 02:13:00.293051 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:00.292828 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bd627785-40f4-4336-b29e-691de0575e7b-service-ca\") pod \"console-5fbbd6f97f-zm5fz\" (UID: \"bd627785-40f4-4336-b29e-691de0575e7b\") " pod="openshift-console/console-5fbbd6f97f-zm5fz" Jul 10 02:13:00.293628 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:00.293456 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bd627785-40f4-4336-b29e-691de0575e7b-console-config\") pod \"console-5fbbd6f97f-zm5fz\" (UID: \"bd627785-40f4-4336-b29e-691de0575e7b\") " pod="openshift-console/console-5fbbd6f97f-zm5fz" Jul 10 02:13:00.293628 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:00.293536 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bd627785-40f4-4336-b29e-691de0575e7b-service-ca\") pod \"console-5fbbd6f97f-zm5fz\" (UID: \"bd627785-40f4-4336-b29e-691de0575e7b\") " pod="openshift-console/console-5fbbd6f97f-zm5fz" Jul 10 02:13:00.293804 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:00.293756 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bd627785-40f4-4336-b29e-691de0575e7b-oauth-serving-cert\") pod \"console-5fbbd6f97f-zm5fz\" (UID: \"bd627785-40f4-4336-b29e-691de0575e7b\") " pod="openshift-console/console-5fbbd6f97f-zm5fz" Jul 10 02:13:00.294018 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:00.293994 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd627785-40f4-4336-b29e-691de0575e7b-trusted-ca-bundle\") pod \"console-5fbbd6f97f-zm5fz\" (UID: \"bd627785-40f4-4336-b29e-691de0575e7b\") " pod="openshift-console/console-5fbbd6f97f-zm5fz" Jul 10 02:13:00.296154 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:00.296128 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bd627785-40f4-4336-b29e-691de0575e7b-console-serving-cert\") pod \"console-5fbbd6f97f-zm5fz\" (UID: \"bd627785-40f4-4336-b29e-691de0575e7b\") " pod="openshift-console/console-5fbbd6f97f-zm5fz" Jul 10 02:13:00.296276 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:00.296193 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bd627785-40f4-4336-b29e-691de0575e7b-console-oauth-config\") pod \"console-5fbbd6f97f-zm5fz\" (UID: \"bd627785-40f4-4336-b29e-691de0575e7b\") " pod="openshift-console/console-5fbbd6f97f-zm5fz" Jul 10 02:13:00.303707 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:00.303665 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25hw8\" (UniqueName: \"kubernetes.io/projected/bd627785-40f4-4336-b29e-691de0575e7b-kube-api-access-25hw8\") pod \"console-5fbbd6f97f-zm5fz\" (UID: \"bd627785-40f4-4336-b29e-691de0575e7b\") " pod="openshift-console/console-5fbbd6f97f-zm5fz" Jul 10 02:13:00.339726 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:00.339671 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fbbd6f97f-zm5fz" Jul 10 02:13:00.547223 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:00.547182 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fbbd6f97f-zm5fz"] Jul 10 02:13:00.550802 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:13:00.550770 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd627785_40f4_4336_b29e_691de0575e7b.slice/crio-2a1bf0d16c230769daea955daae7b79ab2809db3e65e929a136a03596f2bcfb4 WatchSource:0}: Error finding container 2a1bf0d16c230769daea955daae7b79ab2809db3e65e929a136a03596f2bcfb4: Status 404 returned error can't find the container with id 2a1bf0d16c230769daea955daae7b79ab2809db3e65e929a136a03596f2bcfb4 Jul 10 02:13:00.967231 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:00.967142 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-whpts" event={"ID":"54ca438c-c22e-45df-8aa2-f653453022b0","Type":"ContainerStarted","Data":"72d3e93b7ada1e562589c65e5dc94abd8dbed2c8ffd7ea139390fa97ee0be7ca"} Jul 10 02:13:00.969114 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:00.969078 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fbbd6f97f-zm5fz" event={"ID":"bd627785-40f4-4336-b29e-691de0575e7b","Type":"ContainerStarted","Data":"6050327ef4b80ad154edc4d0cb76c1bf91a3de7d2fb713b11fcdaf35a7780318"} Jul 10 02:13:00.969254 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:00.969133 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fbbd6f97f-zm5fz" event={"ID":"bd627785-40f4-4336-b29e-691de0575e7b","Type":"ContainerStarted","Data":"2a1bf0d16c230769daea955daae7b79ab2809db3e65e929a136a03596f2bcfb4"} Jul 10 02:13:04.405458 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:04.405358 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:13:04.988919 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:04.988888 2566 generic.go:356] "Generic (PLEG): container finished" podID="397d53aa-cc3d-4319-8d59-10127a7c6883" containerID="6e45d10e5536049994844a90b9da4166561864f684c97e768f4667986e8f55d2" exitCode=0 Jul 10 02:13:04.989087 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:04.988957 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5z9lh" event={"ID":"397d53aa-cc3d-4319-8d59-10127a7c6883","Type":"ContainerDied","Data":"6e45d10e5536049994844a90b9da4166561864f684c97e768f4667986e8f55d2"} Jul 10 02:13:04.989344 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:04.989322 2566 scope.go:117] "RemoveContainer" containerID="6e45d10e5536049994844a90b9da4166561864f684c97e768f4667986e8f55d2" Jul 10 02:13:05.007183 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:05.007129 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5fbbd6f97f-zm5fz" podStartSLOduration=6.00711013 podStartE2EDuration="6.00711013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:12:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:13:01.005799171 +0000 UTC m=+312.582076256" watchObservedRunningTime="2026-07-10 02:13:05.00711013 +0000 UTC m=+316.583387217" Jul 10 02:13:05.996036 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:05.996001 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5z9lh" event={"ID":"397d53aa-cc3d-4319-8d59-10127a7c6883","Type":"ContainerStarted","Data":"558cd65f576a9ff54ab4c653cc3278ddeca5fbcc383ec83bb4a746c237ee04aa"} Jul 10 02:13:06.599977 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:06.599936 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:13:06.653729 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:06.653684 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:13:08.233083 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:08.233048 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:13:08.303454 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:08.303423 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:13:08.758427 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:08.758392 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d7b8c6c9d-h7w9q" Jul 10 02:13:08.758627 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:08.758450 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d7b8c6c9d-h7w9q" Jul 10 02:13:08.763472 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:08.763450 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d7b8c6c9d-h7w9q" Jul 10 02:13:09.017815 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:09.017730 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d7b8c6c9d-h7w9q" Jul 10 02:13:09.028811 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:09.028781 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:13:09.130032 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:09.129997 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:13:10.340768 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:10.340736 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5fbbd6f97f-zm5fz" Jul 10 02:13:10.341265 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:10.340798 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5fbbd6f97f-zm5fz" Jul 10 02:13:10.346882 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:10.346861 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5fbbd6f97f-zm5fz" Jul 10 02:13:10.746088 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:10.746020 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7bd9d9f484-mmhsv" Jul 10 02:13:10.749814 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:10.749794 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7bd9d9f484-mmhsv" Jul 10 02:13:11.024988 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:11.024962 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5fbbd6f97f-zm5fz" Jul 10 02:13:11.073582 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:11.073556 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d7b8c6c9d-h7w9q"] Jul 10 02:13:12.203724 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:12.203689 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:13:12.258631 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:12.258584 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:13:13.602061 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:13.602024 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:13:18.612262 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:18.612229 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:13:18.666964 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:18.666930 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:13:23.237411 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:23.237373 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:13:23.338841 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:23.338812 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:13:35.220468 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:35.220434 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:13:36.094388 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:36.094345 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5d7b8c6c9d-h7w9q" podUID="5361f125-2d98-409a-8966-f54d6f90066b" containerName="console" containerID="cri-o://189377c7de2d87ba37b378f67af6937179340d0268d295fd11bd6098a817a395" gracePeriod=15 Jul 10 02:13:36.362448 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:36.362393 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d7b8c6c9d-h7w9q" Jul 10 02:13:36.490107 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:36.490075 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5361f125-2d98-409a-8966-f54d6f90066b-console-config\") pod \"5361f125-2d98-409a-8966-f54d6f90066b\" (UID: \"5361f125-2d98-409a-8966-f54d6f90066b\") " Jul 10 02:13:36.490266 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:36.490133 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5361f125-2d98-409a-8966-f54d6f90066b-console-serving-cert\") pod \"5361f125-2d98-409a-8966-f54d6f90066b\" (UID: \"5361f125-2d98-409a-8966-f54d6f90066b\") " Jul 10 02:13:36.490266 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:36.490180 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46hq2\" (UniqueName: \"kubernetes.io/projected/5361f125-2d98-409a-8966-f54d6f90066b-kube-api-access-46hq2\") pod \"5361f125-2d98-409a-8966-f54d6f90066b\" (UID: \"5361f125-2d98-409a-8966-f54d6f90066b\") " Jul 10 02:13:36.490266 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:36.490202 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5361f125-2d98-409a-8966-f54d6f90066b-service-ca\") pod \"5361f125-2d98-409a-8966-f54d6f90066b\" (UID: \"5361f125-2d98-409a-8966-f54d6f90066b\") " Jul 10 02:13:36.490266 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:36.490229 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5361f125-2d98-409a-8966-f54d6f90066b-console-oauth-config\") pod \"5361f125-2d98-409a-8966-f54d6f90066b\" (UID: \"5361f125-2d98-409a-8966-f54d6f90066b\") " Jul 10 02:13:36.490266 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:36.490256 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5361f125-2d98-409a-8966-f54d6f90066b-oauth-serving-cert\") pod \"5361f125-2d98-409a-8966-f54d6f90066b\" (UID: \"5361f125-2d98-409a-8966-f54d6f90066b\") " Jul 10 02:13:36.490655 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:36.490623 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5361f125-2d98-409a-8966-f54d6f90066b-console-config" (OuterVolumeSpecName: "console-config") pod "5361f125-2d98-409a-8966-f54d6f90066b" (UID: "5361f125-2d98-409a-8966-f54d6f90066b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 02:13:36.490747 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:36.490722 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5361f125-2d98-409a-8966-f54d6f90066b-service-ca" (OuterVolumeSpecName: "service-ca") pod "5361f125-2d98-409a-8966-f54d6f90066b" (UID: "5361f125-2d98-409a-8966-f54d6f90066b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 02:13:36.490804 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:36.490738 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5361f125-2d98-409a-8966-f54d6f90066b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5361f125-2d98-409a-8966-f54d6f90066b" (UID: "5361f125-2d98-409a-8966-f54d6f90066b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 02:13:36.492406 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:36.492378 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5361f125-2d98-409a-8966-f54d6f90066b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5361f125-2d98-409a-8966-f54d6f90066b" (UID: "5361f125-2d98-409a-8966-f54d6f90066b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 02:13:36.492532 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:36.492411 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5361f125-2d98-409a-8966-f54d6f90066b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5361f125-2d98-409a-8966-f54d6f90066b" (UID: "5361f125-2d98-409a-8966-f54d6f90066b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 02:13:36.492532 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:36.492476 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5361f125-2d98-409a-8966-f54d6f90066b-kube-api-access-46hq2" (OuterVolumeSpecName: "kube-api-access-46hq2") pod "5361f125-2d98-409a-8966-f54d6f90066b" (UID: "5361f125-2d98-409a-8966-f54d6f90066b"). InnerVolumeSpecName "kube-api-access-46hq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 02:13:36.591395 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:36.591371 2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5361f125-2d98-409a-8966-f54d6f90066b-console-config\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:13:36.591395 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:36.591391 2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5361f125-2d98-409a-8966-f54d6f90066b-console-serving-cert\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:13:36.591520 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:36.591401 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46hq2\" (UniqueName: \"kubernetes.io/projected/5361f125-2d98-409a-8966-f54d6f90066b-kube-api-access-46hq2\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:13:36.591520 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:36.591410 2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5361f125-2d98-409a-8966-f54d6f90066b-service-ca\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:13:36.591520 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:36.591418 2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5361f125-2d98-409a-8966-f54d6f90066b-console-oauth-config\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:13:36.591520 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:36.591425 2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5361f125-2d98-409a-8966-f54d6f90066b-oauth-serving-cert\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:13:37.116085 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:37.116056 2566 generic.go:356] "Generic (PLEG): container finished" podID="5361f125-2d98-409a-8966-f54d6f90066b" containerID="189377c7de2d87ba37b378f67af6937179340d0268d295fd11bd6098a817a395" exitCode=0 Jul 10 02:13:37.116234 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:37.116104 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d7b8c6c9d-h7w9q" Jul 10 02:13:37.116234 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:37.116120 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d7b8c6c9d-h7w9q" event={"ID":"5361f125-2d98-409a-8966-f54d6f90066b","Type":"ContainerDied","Data":"189377c7de2d87ba37b378f67af6937179340d0268d295fd11bd6098a817a395"} Jul 10 02:13:37.116234 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:37.116155 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d7b8c6c9d-h7w9q" event={"ID":"5361f125-2d98-409a-8966-f54d6f90066b","Type":"ContainerDied","Data":"ae625014a7ac1ca6be3c7c0e38d5b4fcc8d57a31d829723e23eabbbf9c1d7594"} Jul 10 02:13:37.116234 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:37.116175 2566 scope.go:117] "RemoveContainer" containerID="189377c7de2d87ba37b378f67af6937179340d0268d295fd11bd6098a817a395" Jul 10 02:13:37.124498 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:37.124480 2566 scope.go:117] "RemoveContainer" containerID="189377c7de2d87ba37b378f67af6937179340d0268d295fd11bd6098a817a395" Jul 10 02:13:37.124824 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:13:37.124801 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"189377c7de2d87ba37b378f67af6937179340d0268d295fd11bd6098a817a395\": container with ID starting with 189377c7de2d87ba37b378f67af6937179340d0268d295fd11bd6098a817a395 not found: ID does not exist" containerID="189377c7de2d87ba37b378f67af6937179340d0268d295fd11bd6098a817a395" Jul 10 02:13:37.124939 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:37.124828 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"189377c7de2d87ba37b378f67af6937179340d0268d295fd11bd6098a817a395"} err="failed to get container status \"189377c7de2d87ba37b378f67af6937179340d0268d295fd11bd6098a817a395\": rpc error: code = NotFound desc = could not find container \"189377c7de2d87ba37b378f67af6937179340d0268d295fd11bd6098a817a395\": container with ID starting with 189377c7de2d87ba37b378f67af6937179340d0268d295fd11bd6098a817a395 not found: ID does not exist" Jul 10 02:13:37.136474 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:37.136455 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d7b8c6c9d-h7w9q"] Jul 10 02:13:37.141004 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:37.140985 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d7b8c6c9d-h7w9q"] Jul 10 02:13:39.009655 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:39.009627 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5361f125-2d98-409a-8966-f54d6f90066b" path="/var/lib/kubelet/pods/5361f125-2d98-409a-8966-f54d6f90066b/volumes" Jul 10 02:13:39.586134 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:39.586104 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:13:39.643439 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:39.643413 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:13:42.255358 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:42.255283 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:13:42.313635 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:42.313577 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:13:54.191905 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:54.191868 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:13:54.264242 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:13:54.264211 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:14:02.749653 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:02.749614 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:14:02.893688 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:02.893654 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bf086af-02b9-43f2-935f-474133fef4cf-metrics-certs\") pod \"network-metrics-daemon-6hfgs\" (UID: \"8bf086af-02b9-43f2-935f-474133fef4cf\") " pod="openshift-multus/network-metrics-daemon-6hfgs" Jul 10 02:14:02.896005 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:02.895987 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bf086af-02b9-43f2-935f-474133fef4cf-metrics-certs\") pod \"network-metrics-daemon-6hfgs\" (UID: \"8bf086af-02b9-43f2-935f-474133fef4cf\") " pod="openshift-multus/network-metrics-daemon-6hfgs" Jul 10 02:14:03.018529 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:03.018455 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dzm46\"" Jul 10 02:14:03.026161 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:03.026138 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hfgs" Jul 10 02:14:03.136144 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:03.136113 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-75c57dfdcc-mbjmf"] Jul 10 02:14:03.136426 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:03.136403 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5361f125-2d98-409a-8966-f54d6f90066b" containerName="console" Jul 10 02:14:03.136426 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:03.136418 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5361f125-2d98-409a-8966-f54d6f90066b" containerName="console" Jul 10 02:14:03.136518 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:03.136479 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5361f125-2d98-409a-8966-f54d6f90066b" containerName="console" Jul 10 02:14:03.138290 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:03.138270 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75c57dfdcc-mbjmf" Jul 10 02:14:03.154842 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:03.154815 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75c57dfdcc-mbjmf"] Jul 10 02:14:03.156408 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:03.156391 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6hfgs"] Jul 10 02:14:03.156590 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:14:03.156568 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bf086af_02b9_43f2_935f_474133fef4cf.slice/crio-f08894f8fd76a916bce30a081d37570751253b393dde206a063f62d3a199695a WatchSource:0}: Error finding container f08894f8fd76a916bce30a081d37570751253b393dde206a063f62d3a199695a: Status 404 returned error can't find the container with id f08894f8fd76a916bce30a081d37570751253b393dde206a063f62d3a199695a Jul 10 02:14:03.158740 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:03.158725 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 02:14:03.212669 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:03.212638 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6hfgs" event={"ID":"8bf086af-02b9-43f2-935f-474133fef4cf","Type":"ContainerStarted","Data":"f08894f8fd76a916bce30a081d37570751253b393dde206a063f62d3a199695a"} Jul 10 02:14:03.217999 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:03.217973 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:14:03.296457 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:03.296430 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a91a4edc-b8be-4cf7-a86d-3177af27d334-console-oauth-config\") pod \"console-75c57dfdcc-mbjmf\" (UID: \"a91a4edc-b8be-4cf7-a86d-3177af27d334\") " pod="openshift-console/console-75c57dfdcc-mbjmf" Jul 10 02:14:03.296559 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:03.296474 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a91a4edc-b8be-4cf7-a86d-3177af27d334-console-config\") pod \"console-75c57dfdcc-mbjmf\" (UID: \"a91a4edc-b8be-4cf7-a86d-3177af27d334\") " pod="openshift-console/console-75c57dfdcc-mbjmf" Jul 10 02:14:03.296559 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:03.296536 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a91a4edc-b8be-4cf7-a86d-3177af27d334-service-ca\") pod \"console-75c57dfdcc-mbjmf\" (UID: \"a91a4edc-b8be-4cf7-a86d-3177af27d334\") " pod="openshift-console/console-75c57dfdcc-mbjmf" Jul 10 02:14:03.296677 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:03.296562 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgx6p\" (UniqueName: \"kubernetes.io/projected/a91a4edc-b8be-4cf7-a86d-3177af27d334-kube-api-access-jgx6p\") pod \"console-75c57dfdcc-mbjmf\" (UID: \"a91a4edc-b8be-4cf7-a86d-3177af27d334\") " pod="openshift-console/console-75c57dfdcc-mbjmf" Jul 10 02:14:03.296677 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:03.296584 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a91a4edc-b8be-4cf7-a86d-3177af27d334-trusted-ca-bundle\") pod \"console-75c57dfdcc-mbjmf\" (UID: \"a91a4edc-b8be-4cf7-a86d-3177af27d334\") " pod="openshift-console/console-75c57dfdcc-mbjmf" Jul 10 02:14:03.296677 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:03.296615 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a91a4edc-b8be-4cf7-a86d-3177af27d334-oauth-serving-cert\") pod \"console-75c57dfdcc-mbjmf\" (UID: \"a91a4edc-b8be-4cf7-a86d-3177af27d334\") " pod="openshift-console/console-75c57dfdcc-mbjmf" Jul 10 02:14:03.296822 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:03.296724 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a91a4edc-b8be-4cf7-a86d-3177af27d334-console-serving-cert\") pod \"console-75c57dfdcc-mbjmf\" (UID: \"a91a4edc-b8be-4cf7-a86d-3177af27d334\") " pod="openshift-console/console-75c57dfdcc-mbjmf" Jul 10 02:14:03.313689 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:03.313667 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:14:03.397778 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:03.397754 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a91a4edc-b8be-4cf7-a86d-3177af27d334-trusted-ca-bundle\") pod \"console-75c57dfdcc-mbjmf\" (UID: \"a91a4edc-b8be-4cf7-a86d-3177af27d334\") " pod="openshift-console/console-75c57dfdcc-mbjmf" Jul 10 02:14:03.397870 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:03.397786 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a91a4edc-b8be-4cf7-a86d-3177af27d334-oauth-serving-cert\") pod \"console-75c57dfdcc-mbjmf\" (UID: \"a91a4edc-b8be-4cf7-a86d-3177af27d334\") " pod="openshift-console/console-75c57dfdcc-mbjmf" Jul 10 02:14:03.397870 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:03.397819 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a91a4edc-b8be-4cf7-a86d-3177af27d334-console-serving-cert\") pod \"console-75c57dfdcc-mbjmf\" (UID: \"a91a4edc-b8be-4cf7-a86d-3177af27d334\") " pod="openshift-console/console-75c57dfdcc-mbjmf" Jul 10 02:14:03.397961 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:03.397885 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a91a4edc-b8be-4cf7-a86d-3177af27d334-console-oauth-config\") pod \"console-75c57dfdcc-mbjmf\" (UID: \"a91a4edc-b8be-4cf7-a86d-3177af27d334\") " pod="openshift-console/console-75c57dfdcc-mbjmf" Jul 10 02:14:03.397961 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:03.397922 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a91a4edc-b8be-4cf7-a86d-3177af27d334-console-config\") pod \"console-75c57dfdcc-mbjmf\" (UID: \"a91a4edc-b8be-4cf7-a86d-3177af27d334\") " pod="openshift-console/console-75c57dfdcc-mbjmf" Jul 10 02:14:03.398068 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:03.397961 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a91a4edc-b8be-4cf7-a86d-3177af27d334-service-ca\") pod \"console-75c57dfdcc-mbjmf\" (UID: \"a91a4edc-b8be-4cf7-a86d-3177af27d334\") " pod="openshift-console/console-75c57dfdcc-mbjmf" Jul 10 02:14:03.398113 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:03.398081 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgx6p\" (UniqueName: \"kubernetes.io/projected/a91a4edc-b8be-4cf7-a86d-3177af27d334-kube-api-access-jgx6p\") pod \"console-75c57dfdcc-mbjmf\" (UID: \"a91a4edc-b8be-4cf7-a86d-3177af27d334\") " pod="openshift-console/console-75c57dfdcc-mbjmf" Jul 10 02:14:03.398676 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:03.398651 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a91a4edc-b8be-4cf7-a86d-3177af27d334-oauth-serving-cert\") pod \"console-75c57dfdcc-mbjmf\" (UID: \"a91a4edc-b8be-4cf7-a86d-3177af27d334\") " pod="openshift-console/console-75c57dfdcc-mbjmf" Jul 10 02:14:03.398758 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:03.398653 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a91a4edc-b8be-4cf7-a86d-3177af27d334-console-config\") pod \"console-75c57dfdcc-mbjmf\" (UID: \"a91a4edc-b8be-4cf7-a86d-3177af27d334\") " pod="openshift-console/console-75c57dfdcc-mbjmf" Jul 10 02:14:03.398809 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:03.398767 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a91a4edc-b8be-4cf7-a86d-3177af27d334-service-ca\") pod \"console-75c57dfdcc-mbjmf\" (UID: \"a91a4edc-b8be-4cf7-a86d-3177af27d334\") " pod="openshift-console/console-75c57dfdcc-mbjmf" Jul 10 02:14:03.398869 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:03.398853 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a91a4edc-b8be-4cf7-a86d-3177af27d334-trusted-ca-bundle\") pod \"console-75c57dfdcc-mbjmf\" (UID: \"a91a4edc-b8be-4cf7-a86d-3177af27d334\") " pod="openshift-console/console-75c57dfdcc-mbjmf" Jul 10 02:14:03.400295 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:03.400278 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a91a4edc-b8be-4cf7-a86d-3177af27d334-console-oauth-config\") pod \"console-75c57dfdcc-mbjmf\" (UID: \"a91a4edc-b8be-4cf7-a86d-3177af27d334\") " pod="openshift-console/console-75c57dfdcc-mbjmf" Jul 10 02:14:03.400528 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:03.400507 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a91a4edc-b8be-4cf7-a86d-3177af27d334-console-serving-cert\") pod \"console-75c57dfdcc-mbjmf\" (UID: \"a91a4edc-b8be-4cf7-a86d-3177af27d334\") " pod="openshift-console/console-75c57dfdcc-mbjmf" Jul 10 02:14:03.408317 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:03.408300 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgx6p\" (UniqueName: \"kubernetes.io/projected/a91a4edc-b8be-4cf7-a86d-3177af27d334-kube-api-access-jgx6p\") pod \"console-75c57dfdcc-mbjmf\" (UID: \"a91a4edc-b8be-4cf7-a86d-3177af27d334\") " pod="openshift-console/console-75c57dfdcc-mbjmf" Jul 10 02:14:03.411336 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:03.411315 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:14:03.448193 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:03.448168 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75c57dfdcc-mbjmf" Jul 10 02:14:03.571501 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:03.571481 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75c57dfdcc-mbjmf"] Jul 10 02:14:03.573481 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:14:03.573452 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda91a4edc_b8be_4cf7_a86d_3177af27d334.slice/crio-323fa33fa95191aa832ffe52a92f295d386edf267b63d484b51c09d83a9b2703 WatchSource:0}: Error finding container 323fa33fa95191aa832ffe52a92f295d386edf267b63d484b51c09d83a9b2703: Status 404 returned error can't find the container with id 323fa33fa95191aa832ffe52a92f295d386edf267b63d484b51c09d83a9b2703 Jul 10 02:14:03.942791 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:03.942696 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:14:04.001550 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:04.001511 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:14:04.221383 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:04.221343 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75c57dfdcc-mbjmf" event={"ID":"a91a4edc-b8be-4cf7-a86d-3177af27d334","Type":"ContainerStarted","Data":"c256828ff37de6ea61f31b346c3c83dcaae0a52e86073df37a869299f8cc2e6f"} Jul 10 02:14:04.221522 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:04.221410 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75c57dfdcc-mbjmf" event={"ID":"a91a4edc-b8be-4cf7-a86d-3177af27d334","Type":"ContainerStarted","Data":"323fa33fa95191aa832ffe52a92f295d386edf267b63d484b51c09d83a9b2703"} Jul 10 02:14:04.223556 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:04.223526 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6hfgs" event={"ID":"8bf086af-02b9-43f2-935f-474133fef4cf","Type":"ContainerStarted","Data":"a1b6f72bcd4280c44b85bfe57705bdc765b959cc93bd721b6142ef7049a8450c"} Jul 10 02:14:04.245302 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:04.245254 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-75c57dfdcc-mbjmf" podStartSLOduration=1.245237818 podStartE2EDuration="1.245237818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:14:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:14:04.243507634 +0000 UTC m=+375.819784721" watchObservedRunningTime="2026-07-10 02:14:04.245237818 +0000 UTC m=+375.821514903" Jul 10 02:14:05.229044 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:05.229005 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6hfgs" event={"ID":"8bf086af-02b9-43f2-935f-474133fef4cf","Type":"ContainerStarted","Data":"2b844250361c6a7bf4c61ed87b38c9ac51c33001fd95c54b24300ef9c69c59e5"} Jul 10 02:14:05.252970 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:05.252928 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6hfgs" podStartSLOduration=375.292624285 podStartE2EDuration="6m16.252915478s" totalImagesPullingTime="960.291193ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:07:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:14:05.252074686 +0000 UTC m=+376.828351791" watchObservedRunningTime="2026-07-10 02:14:05.252915478 +0000 UTC m=+376.829192564" Jul 10 02:14:05.358225 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:05.358189 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:14:05.450209 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:05.450177 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:14:08.239817 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:08.239782 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:14:08.339500 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:08.339458 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:14:08.399245 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:08.399208 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:14:13.449159 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:13.449122 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-75c57dfdcc-mbjmf" Jul 10 02:14:13.449567 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:13.449210 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75c57dfdcc-mbjmf" Jul 10 02:14:13.453612 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:13.453585 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-75c57dfdcc-mbjmf" Jul 10 02:14:14.268105 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:14.268074 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-75c57dfdcc-mbjmf" Jul 10 02:14:14.324165 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:14.324135 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5fbbd6f97f-zm5fz"] Jul 10 02:14:14.446155 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:14.446115 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:14:14.546743 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:14.546711 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:14:14.602159 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:14.602120 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:14:39.348835 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:39.348786 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5fbbd6f97f-zm5fz" podUID="bd627785-40f4-4336-b29e-691de0575e7b" containerName="console" containerID="cri-o://6050327ef4b80ad154edc4d0cb76c1bf91a3de7d2fb713b11fcdaf35a7780318" gracePeriod=15 Jul 10 02:14:39.578177 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:39.578140 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fbbd6f97f-zm5fz" Jul 10 02:14:39.661686 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:39.661618 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25hw8\" (UniqueName: \"kubernetes.io/projected/bd627785-40f4-4336-b29e-691de0575e7b-kube-api-access-25hw8\") pod \"bd627785-40f4-4336-b29e-691de0575e7b\" (UID: \"bd627785-40f4-4336-b29e-691de0575e7b\") " Jul 10 02:14:39.661686 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:39.661672 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bd627785-40f4-4336-b29e-691de0575e7b-service-ca\") pod \"bd627785-40f4-4336-b29e-691de0575e7b\" (UID: \"bd627785-40f4-4336-b29e-691de0575e7b\") " Jul 10 02:14:39.661879 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:39.661703 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bd627785-40f4-4336-b29e-691de0575e7b-oauth-serving-cert\") pod \"bd627785-40f4-4336-b29e-691de0575e7b\" (UID: \"bd627785-40f4-4336-b29e-691de0575e7b\") " Jul 10 02:14:39.661879 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:39.661719 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bd627785-40f4-4336-b29e-691de0575e7b-console-config\") pod \"bd627785-40f4-4336-b29e-691de0575e7b\" (UID: \"bd627785-40f4-4336-b29e-691de0575e7b\") " Jul 10 02:14:39.661879 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:39.661761 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd627785-40f4-4336-b29e-691de0575e7b-trusted-ca-bundle\") pod \"bd627785-40f4-4336-b29e-691de0575e7b\" (UID: \"bd627785-40f4-4336-b29e-691de0575e7b\") " Jul 10 02:14:39.661879 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:39.661778 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bd627785-40f4-4336-b29e-691de0575e7b-console-oauth-config\") pod \"bd627785-40f4-4336-b29e-691de0575e7b\" (UID: \"bd627785-40f4-4336-b29e-691de0575e7b\") " Jul 10 02:14:39.661879 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:39.661819 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bd627785-40f4-4336-b29e-691de0575e7b-console-serving-cert\") pod \"bd627785-40f4-4336-b29e-691de0575e7b\" (UID: \"bd627785-40f4-4336-b29e-691de0575e7b\") " Jul 10 02:14:39.662166 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:39.662134 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd627785-40f4-4336-b29e-691de0575e7b-service-ca" (OuterVolumeSpecName: "service-ca") pod "bd627785-40f4-4336-b29e-691de0575e7b" (UID: "bd627785-40f4-4336-b29e-691de0575e7b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 02:14:39.662166 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:39.662144 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd627785-40f4-4336-b29e-691de0575e7b-console-config" (OuterVolumeSpecName: "console-config") pod "bd627785-40f4-4336-b29e-691de0575e7b" (UID: "bd627785-40f4-4336-b29e-691de0575e7b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 02:14:39.662282 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:39.662153 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd627785-40f4-4336-b29e-691de0575e7b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "bd627785-40f4-4336-b29e-691de0575e7b" (UID: "bd627785-40f4-4336-b29e-691de0575e7b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 02:14:39.662375 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:39.662354 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd627785-40f4-4336-b29e-691de0575e7b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "bd627785-40f4-4336-b29e-691de0575e7b" (UID: "bd627785-40f4-4336-b29e-691de0575e7b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 02:14:39.663837 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:39.663813 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd627785-40f4-4336-b29e-691de0575e7b-kube-api-access-25hw8" (OuterVolumeSpecName: "kube-api-access-25hw8") pod "bd627785-40f4-4336-b29e-691de0575e7b" (UID: "bd627785-40f4-4336-b29e-691de0575e7b"). InnerVolumeSpecName "kube-api-access-25hw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 02:14:39.663921 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:39.663871 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd627785-40f4-4336-b29e-691de0575e7b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "bd627785-40f4-4336-b29e-691de0575e7b" (UID: "bd627785-40f4-4336-b29e-691de0575e7b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 02:14:39.663921 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:39.663887 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd627785-40f4-4336-b29e-691de0575e7b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "bd627785-40f4-4336-b29e-691de0575e7b" (UID: "bd627785-40f4-4336-b29e-691de0575e7b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 02:14:39.763273 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:39.763236 2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bd627785-40f4-4336-b29e-691de0575e7b-service-ca\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:14:39.763273 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:39.763270 2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bd627785-40f4-4336-b29e-691de0575e7b-oauth-serving-cert\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:14:39.763273 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:39.763279 2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bd627785-40f4-4336-b29e-691de0575e7b-console-config\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:14:39.763273 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:39.763288 2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd627785-40f4-4336-b29e-691de0575e7b-trusted-ca-bundle\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:14:39.763508 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:39.763295 2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bd627785-40f4-4336-b29e-691de0575e7b-console-oauth-config\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:14:39.763508 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:39.763304 2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bd627785-40f4-4336-b29e-691de0575e7b-console-serving-cert\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:14:39.763508 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:39.763312 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25hw8\" (UniqueName: \"kubernetes.io/projected/bd627785-40f4-4336-b29e-691de0575e7b-kube-api-access-25hw8\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:14:40.366259 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:40.366221 2566 generic.go:356] "Generic (PLEG): container finished" podID="bd627785-40f4-4336-b29e-691de0575e7b" containerID="6050327ef4b80ad154edc4d0cb76c1bf91a3de7d2fb713b11fcdaf35a7780318" exitCode=0 Jul 10 02:14:40.366675 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:40.366266 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fbbd6f97f-zm5fz" Jul 10 02:14:40.366675 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:40.366310 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fbbd6f97f-zm5fz" event={"ID":"bd627785-40f4-4336-b29e-691de0575e7b","Type":"ContainerDied","Data":"6050327ef4b80ad154edc4d0cb76c1bf91a3de7d2fb713b11fcdaf35a7780318"} Jul 10 02:14:40.366675 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:40.366367 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fbbd6f97f-zm5fz" event={"ID":"bd627785-40f4-4336-b29e-691de0575e7b","Type":"ContainerDied","Data":"2a1bf0d16c230769daea955daae7b79ab2809db3e65e929a136a03596f2bcfb4"} Jul 10 02:14:40.366675 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:40.366387 2566 scope.go:117] "RemoveContainer" containerID="6050327ef4b80ad154edc4d0cb76c1bf91a3de7d2fb713b11fcdaf35a7780318" Jul 10 02:14:40.375713 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:40.375138 2566 scope.go:117] "RemoveContainer" containerID="6050327ef4b80ad154edc4d0cb76c1bf91a3de7d2fb713b11fcdaf35a7780318" Jul 10 02:14:40.375713 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:14:40.375421 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6050327ef4b80ad154edc4d0cb76c1bf91a3de7d2fb713b11fcdaf35a7780318\": container with ID starting with 6050327ef4b80ad154edc4d0cb76c1bf91a3de7d2fb713b11fcdaf35a7780318 not found: ID does not exist" containerID="6050327ef4b80ad154edc4d0cb76c1bf91a3de7d2fb713b11fcdaf35a7780318" Jul 10 02:14:40.375713 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:40.375457 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6050327ef4b80ad154edc4d0cb76c1bf91a3de7d2fb713b11fcdaf35a7780318"} err="failed to get container status \"6050327ef4b80ad154edc4d0cb76c1bf91a3de7d2fb713b11fcdaf35a7780318\": rpc error: code = NotFound desc = could not find container \"6050327ef4b80ad154edc4d0cb76c1bf91a3de7d2fb713b11fcdaf35a7780318\": container with ID starting with 6050327ef4b80ad154edc4d0cb76c1bf91a3de7d2fb713b11fcdaf35a7780318 not found: ID does not exist" Jul 10 02:14:40.390630 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:40.390589 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5fbbd6f97f-zm5fz"] Jul 10 02:14:40.395117 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:40.395097 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5fbbd6f97f-zm5fz"] Jul 10 02:14:41.010565 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:41.010533 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd627785-40f4-4336-b29e-691de0575e7b" path="/var/lib/kubelet/pods/bd627785-40f4-4336-b29e-691de0575e7b/volumes" Jul 10 02:14:46.026178 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:46.026111 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:14:46.128659 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:46.128587 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:14:46.757462 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:46.757430 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vj867"] Jul 10 02:14:46.757780 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:46.757767 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd627785-40f4-4336-b29e-691de0575e7b" containerName="console" Jul 10 02:14:46.757836 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:46.757781 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd627785-40f4-4336-b29e-691de0575e7b" containerName="console" Jul 10 02:14:46.757836 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:46.757829 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd627785-40f4-4336-b29e-691de0575e7b" containerName="console" Jul 10 02:14:46.759861 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:46.759843 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vj867" Jul 10 02:14:46.762903 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:46.762880 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 10 02:14:46.764296 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:46.764275 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 10 02:14:46.764343 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:46.764309 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-mt7hs\"" Jul 10 02:14:46.770002 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:46.769968 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vj867"] Jul 10 02:14:46.816366 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:46.816337 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x258\" (UniqueName: \"kubernetes.io/projected/2017559a-9cee-4017-b92b-8567948c2ebf-kube-api-access-2x258\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vj867\" (UID: \"2017559a-9cee-4017-b92b-8567948c2ebf\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vj867" Jul 10 02:14:46.816504 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:46.816372 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2017559a-9cee-4017-b92b-8567948c2ebf-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vj867\" (UID: \"2017559a-9cee-4017-b92b-8567948c2ebf\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vj867" Jul 10 02:14:46.816504 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:46.816393 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2017559a-9cee-4017-b92b-8567948c2ebf-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vj867\" (UID: \"2017559a-9cee-4017-b92b-8567948c2ebf\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vj867" Jul 10 02:14:46.917187 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:46.917161 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x258\" (UniqueName: \"kubernetes.io/projected/2017559a-9cee-4017-b92b-8567948c2ebf-kube-api-access-2x258\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vj867\" (UID: \"2017559a-9cee-4017-b92b-8567948c2ebf\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vj867" Jul 10 02:14:46.917311 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:46.917197 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2017559a-9cee-4017-b92b-8567948c2ebf-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vj867\" (UID: \"2017559a-9cee-4017-b92b-8567948c2ebf\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vj867" Jul 10 02:14:46.917311 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:46.917216 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2017559a-9cee-4017-b92b-8567948c2ebf-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vj867\" (UID: \"2017559a-9cee-4017-b92b-8567948c2ebf\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vj867" Jul 10 02:14:46.917545 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:46.917528 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2017559a-9cee-4017-b92b-8567948c2ebf-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vj867\" (UID: \"2017559a-9cee-4017-b92b-8567948c2ebf\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vj867" Jul 10 02:14:46.917584 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:46.917563 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2017559a-9cee-4017-b92b-8567948c2ebf-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vj867\" (UID: \"2017559a-9cee-4017-b92b-8567948c2ebf\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vj867" Jul 10 02:14:46.928070 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:46.928038 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x258\" (UniqueName: \"kubernetes.io/projected/2017559a-9cee-4017-b92b-8567948c2ebf-kube-api-access-2x258\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vj867\" (UID: \"2017559a-9cee-4017-b92b-8567948c2ebf\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vj867" Jul 10 02:14:47.069964 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:47.069945 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vj867" Jul 10 02:14:47.196354 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:47.196336 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vj867"] Jul 10 02:14:47.198769 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:14:47.198742 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2017559a_9cee_4017_b92b_8567948c2ebf.slice/crio-f9b1d5a69e4023fa951c392b1d266124962443f1b458d74f2f41948501341f08 WatchSource:0}: Error finding container f9b1d5a69e4023fa951c392b1d266124962443f1b458d74f2f41948501341f08: Status 404 returned error can't find the container with id f9b1d5a69e4023fa951c392b1d266124962443f1b458d74f2f41948501341f08 Jul 10 02:14:47.395372 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:47.395292 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vj867" event={"ID":"2017559a-9cee-4017-b92b-8567948c2ebf","Type":"ContainerStarted","Data":"f9b1d5a69e4023fa951c392b1d266124962443f1b458d74f2f41948501341f08"} Jul 10 02:14:54.426478 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:54.426437 2566 generic.go:356] "Generic (PLEG): container finished" podID="2017559a-9cee-4017-b92b-8567948c2ebf" containerID="abc552dbb935f7cf5726305c65b6c1648cc162046d160e4dfbfd4a3a78a24913" exitCode=0 Jul 10 02:14:54.426862 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:54.426525 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vj867" event={"ID":"2017559a-9cee-4017-b92b-8567948c2ebf","Type":"ContainerDied","Data":"abc552dbb935f7cf5726305c65b6c1648cc162046d160e4dfbfd4a3a78a24913"} Jul 10 02:14:56.120555 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:56.120519 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:14:57.442883 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:57.442848 2566 generic.go:356] "Generic (PLEG): container finished" podID="2017559a-9cee-4017-b92b-8567948c2ebf" containerID="0b6952270e43ff81328677895696743439f04d521efb1be52dcb655bf9c9617b" exitCode=0 Jul 10 02:14:57.443313 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:14:57.442940 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vj867" event={"ID":"2017559a-9cee-4017-b92b-8567948c2ebf","Type":"ContainerDied","Data":"0b6952270e43ff81328677895696743439f04d521efb1be52dcb655bf9c9617b"} Jul 10 02:15:08.488608 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:08.488569 2566 generic.go:356] "Generic (PLEG): container finished" podID="2017559a-9cee-4017-b92b-8567948c2ebf" containerID="bb8aa5d9335a342be241b478a1e9c68adb05c39fabb385a6ec095bf506de33ed" exitCode=0 Jul 10 02:15:08.488996 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:08.488644 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vj867" event={"ID":"2017559a-9cee-4017-b92b-8567948c2ebf","Type":"ContainerDied","Data":"bb8aa5d9335a342be241b478a1e9c68adb05c39fabb385a6ec095bf506de33ed"} Jul 10 02:15:09.614270 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:09.614242 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vj867" Jul 10 02:15:09.701928 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:09.701901 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2017559a-9cee-4017-b92b-8567948c2ebf-util\") pod \"2017559a-9cee-4017-b92b-8567948c2ebf\" (UID: \"2017559a-9cee-4017-b92b-8567948c2ebf\") " Jul 10 02:15:09.702049 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:09.701969 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2x258\" (UniqueName: \"kubernetes.io/projected/2017559a-9cee-4017-b92b-8567948c2ebf-kube-api-access-2x258\") pod \"2017559a-9cee-4017-b92b-8567948c2ebf\" (UID: \"2017559a-9cee-4017-b92b-8567948c2ebf\") " Jul 10 02:15:09.702049 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:09.702023 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2017559a-9cee-4017-b92b-8567948c2ebf-bundle\") pod \"2017559a-9cee-4017-b92b-8567948c2ebf\" (UID: \"2017559a-9cee-4017-b92b-8567948c2ebf\") " Jul 10 02:15:09.702718 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:09.702688 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2017559a-9cee-4017-b92b-8567948c2ebf-bundle" (OuterVolumeSpecName: "bundle") pod "2017559a-9cee-4017-b92b-8567948c2ebf" (UID: "2017559a-9cee-4017-b92b-8567948c2ebf"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 02:15:09.704129 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:09.704111 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2017559a-9cee-4017-b92b-8567948c2ebf-kube-api-access-2x258" (OuterVolumeSpecName: "kube-api-access-2x258") pod "2017559a-9cee-4017-b92b-8567948c2ebf" (UID: "2017559a-9cee-4017-b92b-8567948c2ebf"). InnerVolumeSpecName "kube-api-access-2x258". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 02:15:09.706106 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:09.706084 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2017559a-9cee-4017-b92b-8567948c2ebf-util" (OuterVolumeSpecName: "util") pod "2017559a-9cee-4017-b92b-8567948c2ebf" (UID: "2017559a-9cee-4017-b92b-8567948c2ebf"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 02:15:09.803129 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:09.803102 2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2017559a-9cee-4017-b92b-8567948c2ebf-util\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:15:09.803129 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:09.803126 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x258\" (UniqueName: \"kubernetes.io/projected/2017559a-9cee-4017-b92b-8567948c2ebf-kube-api-access-2x258\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:15:09.803294 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:09.803136 2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2017559a-9cee-4017-b92b-8567948c2ebf-bundle\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:15:10.497833 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:10.497803 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vj867" Jul 10 02:15:10.498065 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:10.497828 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vj867" event={"ID":"2017559a-9cee-4017-b92b-8567948c2ebf","Type":"ContainerDied","Data":"f9b1d5a69e4023fa951c392b1d266124962443f1b458d74f2f41948501341f08"} Jul 10 02:15:10.498065 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:10.497884 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9b1d5a69e4023fa951c392b1d266124962443f1b458d74f2f41948501341f08" Jul 10 02:15:12.804179 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:12.804149 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:15:13.584788 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:13.584746 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:15:14.775652 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:14.775621 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4jnxm"] Jul 10 02:15:14.776123 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:14.775908 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2017559a-9cee-4017-b92b-8567948c2ebf" containerName="util" Jul 10 02:15:14.776123 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:14.775921 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2017559a-9cee-4017-b92b-8567948c2ebf" containerName="util" Jul 10 02:15:14.776123 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:14.775948 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2017559a-9cee-4017-b92b-8567948c2ebf" containerName="extract" Jul 10 02:15:14.776123 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:14.775956 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2017559a-9cee-4017-b92b-8567948c2ebf" containerName="extract" Jul 10 02:15:14.776123 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:14.775972 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2017559a-9cee-4017-b92b-8567948c2ebf" containerName="pull" Jul 10 02:15:14.776123 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:14.775981 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2017559a-9cee-4017-b92b-8567948c2ebf" containerName="pull" Jul 10 02:15:14.776123 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:14.776044 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2017559a-9cee-4017-b92b-8567948c2ebf" containerName="extract" Jul 10 02:15:14.777814 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:14.777797 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4jnxm" Jul 10 02:15:14.781267 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:14.781243 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-5pcgg\"" Jul 10 02:15:14.781542 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:14.781525 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 10 02:15:14.781653 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:14.781562 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 10 02:15:14.794473 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:14.794453 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4jnxm"] Jul 10 02:15:14.943929 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:14.943895 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50108e59-f369-4a85-b64c-346b81f7c4ca-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-4jnxm\" (UID: \"50108e59-f369-4a85-b64c-346b81f7c4ca\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4jnxm" Jul 10 02:15:14.944109 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:14.943940 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fhlr\" (UniqueName: \"kubernetes.io/projected/50108e59-f369-4a85-b64c-346b81f7c4ca-kube-api-access-4fhlr\") pod \"cert-manager-operator-controller-manager-55c444bf86-4jnxm\" (UID: \"50108e59-f369-4a85-b64c-346b81f7c4ca\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4jnxm" Jul 10 02:15:15.044977 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:15.044890 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50108e59-f369-4a85-b64c-346b81f7c4ca-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-4jnxm\" (UID: \"50108e59-f369-4a85-b64c-346b81f7c4ca\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4jnxm" Jul 10 02:15:15.044977 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:15.044944 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fhlr\" (UniqueName: \"kubernetes.io/projected/50108e59-f369-4a85-b64c-346b81f7c4ca-kube-api-access-4fhlr\") pod \"cert-manager-operator-controller-manager-55c444bf86-4jnxm\" (UID: \"50108e59-f369-4a85-b64c-346b81f7c4ca\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4jnxm" Jul 10 02:15:15.045302 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:15.045283 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50108e59-f369-4a85-b64c-346b81f7c4ca-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-4jnxm\" (UID: \"50108e59-f369-4a85-b64c-346b81f7c4ca\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4jnxm" Jul 10 02:15:15.057565 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:15.057535 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fhlr\" (UniqueName: \"kubernetes.io/projected/50108e59-f369-4a85-b64c-346b81f7c4ca-kube-api-access-4fhlr\") pod \"cert-manager-operator-controller-manager-55c444bf86-4jnxm\" (UID: \"50108e59-f369-4a85-b64c-346b81f7c4ca\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4jnxm" Jul 10 02:15:15.087330 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:15.087302 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4jnxm" Jul 10 02:15:15.221835 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:15.221805 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4jnxm"] Jul 10 02:15:15.224458 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:15:15.224425 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50108e59_f369_4a85_b64c_346b81f7c4ca.slice/crio-77382961496ff33031e7d575805d337144e66cacdcdc34df31b99f588cd0292e WatchSource:0}: Error finding container 77382961496ff33031e7d575805d337144e66cacdcdc34df31b99f588cd0292e: Status 404 returned error can't find the container with id 77382961496ff33031e7d575805d337144e66cacdcdc34df31b99f588cd0292e Jul 10 02:15:15.518658 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:15.518551 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4jnxm" event={"ID":"50108e59-f369-4a85-b64c-346b81f7c4ca","Type":"ContainerStarted","Data":"77382961496ff33031e7d575805d337144e66cacdcdc34df31b99f588cd0292e"} Jul 10 02:15:16.413424 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:16.413391 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:15:16.513815 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:16.513760 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:15:18.533469 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:18.533435 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4jnxm" event={"ID":"50108e59-f369-4a85-b64c-346b81f7c4ca","Type":"ContainerStarted","Data":"050988a494a2d79e89224ab0690cd8b1a3135f2fa3be1b78c84601546591da06"} Jul 10 02:15:18.560432 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:18.560374 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4jnxm" podStartSLOduration=2.065559533 podStartE2EDuration="4.560354373s" totalImagesPullingTime="2.49479484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:15:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:15:18.557762548 +0000 UTC m=+450.134039657" watchObservedRunningTime="2026-07-10 02:15:18.560354373 +0000 UTC m=+450.136631460" Jul 10 02:15:18.737668 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:18.737636 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:15:18.839534 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:18.839426 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:15:18.979550 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:18.979516 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:15:19.085003 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:19.084963 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:15:19.111532 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:19.111443 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngqgf"] Jul 10 02:15:19.113855 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:19.113831 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngqgf" Jul 10 02:15:19.116853 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:19.116828 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-mt7hs\"" Jul 10 02:15:19.116951 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:19.116835 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 10 02:15:19.118215 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:19.118199 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 10 02:15:19.130075 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:19.130053 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngqgf"] Jul 10 02:15:19.278214 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:19.278172 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d6ea928e-6166-48a9-873d-f5de8a517d75-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngqgf\" (UID: \"d6ea928e-6166-48a9-873d-f5de8a517d75\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngqgf" Jul 10 02:15:19.278381 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:19.278258 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8hxk\" (UniqueName: \"kubernetes.io/projected/d6ea928e-6166-48a9-873d-f5de8a517d75-kube-api-access-x8hxk\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngqgf\" (UID: \"d6ea928e-6166-48a9-873d-f5de8a517d75\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngqgf" Jul 10 02:15:19.278381 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:19.278298 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d6ea928e-6166-48a9-873d-f5de8a517d75-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngqgf\" (UID: \"d6ea928e-6166-48a9-873d-f5de8a517d75\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngqgf" Jul 10 02:15:19.379473 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:19.379393 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8hxk\" (UniqueName: \"kubernetes.io/projected/d6ea928e-6166-48a9-873d-f5de8a517d75-kube-api-access-x8hxk\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngqgf\" (UID: \"d6ea928e-6166-48a9-873d-f5de8a517d75\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngqgf" Jul 10 02:15:19.379473 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:19.379448 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d6ea928e-6166-48a9-873d-f5de8a517d75-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngqgf\" (UID: \"d6ea928e-6166-48a9-873d-f5de8a517d75\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngqgf" Jul 10 02:15:19.379674 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:19.379483 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d6ea928e-6166-48a9-873d-f5de8a517d75-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngqgf\" (UID: \"d6ea928e-6166-48a9-873d-f5de8a517d75\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngqgf" Jul 10 02:15:19.379957 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:19.379934 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d6ea928e-6166-48a9-873d-f5de8a517d75-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngqgf\" (UID: \"d6ea928e-6166-48a9-873d-f5de8a517d75\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngqgf" Jul 10 02:15:19.379992 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:19.379942 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d6ea928e-6166-48a9-873d-f5de8a517d75-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngqgf\" (UID: \"d6ea928e-6166-48a9-873d-f5de8a517d75\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngqgf" Jul 10 02:15:19.393294 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:19.393267 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8hxk\" (UniqueName: \"kubernetes.io/projected/d6ea928e-6166-48a9-873d-f5de8a517d75-kube-api-access-x8hxk\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngqgf\" (UID: \"d6ea928e-6166-48a9-873d-f5de8a517d75\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngqgf" Jul 10 02:15:19.423425 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:19.423402 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngqgf" Jul 10 02:15:19.549978 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:19.549955 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngqgf"] Jul 10 02:15:19.552097 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:15:19.552058 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6ea928e_6166_48a9_873d_f5de8a517d75.slice/crio-777fe89ef0d0e50cfcf19048440efef89e2dbb117681dd8bf0248e5c2b5bf990 WatchSource:0}: Error finding container 777fe89ef0d0e50cfcf19048440efef89e2dbb117681dd8bf0248e5c2b5bf990: Status 404 returned error can't find the container with id 777fe89ef0d0e50cfcf19048440efef89e2dbb117681dd8bf0248e5c2b5bf990 Jul 10 02:15:20.542685 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:20.542656 2566 generic.go:356] "Generic (PLEG): container finished" podID="d6ea928e-6166-48a9-873d-f5de8a517d75" containerID="eea7104161f14f7be89d4b45b938a0aa116d9077f05836203ae12fa2f8f21f30" exitCode=0 Jul 10 02:15:20.542859 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:20.542748 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngqgf" event={"ID":"d6ea928e-6166-48a9-873d-f5de8a517d75","Type":"ContainerDied","Data":"eea7104161f14f7be89d4b45b938a0aa116d9077f05836203ae12fa2f8f21f30"} Jul 10 02:15:20.542859 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:20.542810 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngqgf" event={"ID":"d6ea928e-6166-48a9-873d-f5de8a517d75","Type":"ContainerStarted","Data":"777fe89ef0d0e50cfcf19048440efef89e2dbb117681dd8bf0248e5c2b5bf990"} Jul 10 02:15:20.954299 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:20.954258 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:15:21.316436 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:21.316349 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-xlznj"] Jul 10 02:15:21.318923 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:21.318897 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-xlznj" Jul 10 02:15:21.322034 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:21.322012 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 10 02:15:21.322146 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:21.322035 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 10 02:15:21.323312 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:21.323294 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-dfck4\"" Jul 10 02:15:21.329513 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:21.329492 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-xlznj"] Jul 10 02:15:21.498072 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:21.498045 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/254106ce-ff9e-4fdf-b528-6ab449733f07-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-xlznj\" (UID: \"254106ce-ff9e-4fdf-b528-6ab449733f07\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xlznj" Jul 10 02:15:21.498247 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:21.498107 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88j2d\" (UniqueName: \"kubernetes.io/projected/254106ce-ff9e-4fdf-b528-6ab449733f07-kube-api-access-88j2d\") pod \"cert-manager-webhook-6486ffd997-xlznj\" (UID: \"254106ce-ff9e-4fdf-b528-6ab449733f07\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xlznj" Jul 10 02:15:21.598686 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:21.598582 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/254106ce-ff9e-4fdf-b528-6ab449733f07-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-xlznj\" (UID: \"254106ce-ff9e-4fdf-b528-6ab449733f07\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xlznj" Jul 10 02:15:21.598686 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:21.598685 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88j2d\" (UniqueName: \"kubernetes.io/projected/254106ce-ff9e-4fdf-b528-6ab449733f07-kube-api-access-88j2d\") pod \"cert-manager-webhook-6486ffd997-xlznj\" (UID: \"254106ce-ff9e-4fdf-b528-6ab449733f07\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xlznj" Jul 10 02:15:21.606293 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:21.606147 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:15:21.608464 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:21.608428 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/254106ce-ff9e-4fdf-b528-6ab449733f07-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-xlznj\" (UID: \"254106ce-ff9e-4fdf-b528-6ab449733f07\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xlznj" Jul 10 02:15:21.608735 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:21.608710 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88j2d\" (UniqueName: \"kubernetes.io/projected/254106ce-ff9e-4fdf-b528-6ab449733f07-kube-api-access-88j2d\") pod \"cert-manager-webhook-6486ffd997-xlznj\" (UID: \"254106ce-ff9e-4fdf-b528-6ab449733f07\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xlznj" Jul 10 02:15:21.641403 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:21.641382 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-xlznj" Jul 10 02:15:21.784945 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:21.784835 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-xlznj"] Jul 10 02:15:22.377220 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:15:22.377189 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod254106ce_ff9e_4fdf_b528_6ab449733f07.slice/crio-b4811e05f810568069ba9387fc7bea8ae7daf42d4d4f84315ba71d8a040a499f WatchSource:0}: Error finding container b4811e05f810568069ba9387fc7bea8ae7daf42d4d4f84315ba71d8a040a499f: Status 404 returned error can't find the container with id b4811e05f810568069ba9387fc7bea8ae7daf42d4d4f84315ba71d8a040a499f Jul 10 02:15:22.554269 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:22.554238 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-xlznj" event={"ID":"254106ce-ff9e-4fdf-b528-6ab449733f07","Type":"ContainerStarted","Data":"b4811e05f810568069ba9387fc7bea8ae7daf42d4d4f84315ba71d8a040a499f"} Jul 10 02:15:22.556099 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:22.556075 2566 generic.go:356] "Generic (PLEG): container finished" podID="d6ea928e-6166-48a9-873d-f5de8a517d75" containerID="32a755e4223f8f3558d5646813cd6dbcc8f7bf823be12033504fc9dd5d768c6f" exitCode=0 Jul 10 02:15:22.556196 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:22.556141 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngqgf" event={"ID":"d6ea928e-6166-48a9-873d-f5de8a517d75","Type":"ContainerDied","Data":"32a755e4223f8f3558d5646813cd6dbcc8f7bf823be12033504fc9dd5d768c6f"} Jul 10 02:15:22.566126 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:22.566085 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:15:22.670415 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:22.670342 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:15:23.414466 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:23.414428 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:15:23.515310 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:23.515269 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:15:23.562436 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:23.562407 2566 generic.go:356] "Generic (PLEG): container finished" podID="d6ea928e-6166-48a9-873d-f5de8a517d75" containerID="7f0211e4cbb8dac44a413ab0a5f50705d67083d28756eae36e561edfdac1e942" exitCode=0 Jul 10 02:15:23.562588 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:23.562453 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngqgf" event={"ID":"d6ea928e-6166-48a9-873d-f5de8a517d75","Type":"ContainerDied","Data":"7f0211e4cbb8dac44a413ab0a5f50705d67083d28756eae36e561edfdac1e942"} Jul 10 02:15:24.687721 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:24.687699 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngqgf" Jul 10 02:15:24.824402 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:24.824307 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x8hxk\" (UniqueName: \"kubernetes.io/projected/d6ea928e-6166-48a9-873d-f5de8a517d75-kube-api-access-x8hxk\") pod \"d6ea928e-6166-48a9-873d-f5de8a517d75\" (UID: \"d6ea928e-6166-48a9-873d-f5de8a517d75\") " Jul 10 02:15:24.824402 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:24.824394 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d6ea928e-6166-48a9-873d-f5de8a517d75-bundle\") pod \"d6ea928e-6166-48a9-873d-f5de8a517d75\" (UID: \"d6ea928e-6166-48a9-873d-f5de8a517d75\") " Jul 10 02:15:24.824586 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:24.824426 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d6ea928e-6166-48a9-873d-f5de8a517d75-util\") pod \"d6ea928e-6166-48a9-873d-f5de8a517d75\" (UID: \"d6ea928e-6166-48a9-873d-f5de8a517d75\") " Jul 10 02:15:24.824817 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:24.824792 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ea928e-6166-48a9-873d-f5de8a517d75-bundle" (OuterVolumeSpecName: "bundle") pod "d6ea928e-6166-48a9-873d-f5de8a517d75" (UID: "d6ea928e-6166-48a9-873d-f5de8a517d75"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 02:15:24.826430 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:24.826409 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6ea928e-6166-48a9-873d-f5de8a517d75-kube-api-access-x8hxk" (OuterVolumeSpecName: "kube-api-access-x8hxk") pod "d6ea928e-6166-48a9-873d-f5de8a517d75" (UID: "d6ea928e-6166-48a9-873d-f5de8a517d75"). InnerVolumeSpecName "kube-api-access-x8hxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 02:15:24.832310 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:24.832285 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ea928e-6166-48a9-873d-f5de8a517d75-util" (OuterVolumeSpecName: "util") pod "d6ea928e-6166-48a9-873d-f5de8a517d75" (UID: "d6ea928e-6166-48a9-873d-f5de8a517d75"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 02:15:24.925782 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:24.925749 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8hxk\" (UniqueName: \"kubernetes.io/projected/d6ea928e-6166-48a9-873d-f5de8a517d75-kube-api-access-x8hxk\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:15:24.925782 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:24.925777 2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d6ea928e-6166-48a9-873d-f5de8a517d75-bundle\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:15:24.925782 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:24.925787 2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d6ea928e-6166-48a9-873d-f5de8a517d75-util\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:15:25.573015 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:25.572982 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngqgf" event={"ID":"d6ea928e-6166-48a9-873d-f5de8a517d75","Type":"ContainerDied","Data":"777fe89ef0d0e50cfcf19048440efef89e2dbb117681dd8bf0248e5c2b5bf990"} Jul 10 02:15:25.573015 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:25.573013 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngqgf" Jul 10 02:15:25.573216 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:25.573029 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="777fe89ef0d0e50cfcf19048440efef89e2dbb117681dd8bf0248e5c2b5bf990" Jul 10 02:15:25.727420 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:25.727382 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-p64sc"] Jul 10 02:15:25.727949 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:25.727852 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6ea928e-6166-48a9-873d-f5de8a517d75" containerName="util" Jul 10 02:15:25.727949 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:25.727871 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6ea928e-6166-48a9-873d-f5de8a517d75" containerName="util" Jul 10 02:15:25.727949 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:25.727882 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6ea928e-6166-48a9-873d-f5de8a517d75" containerName="pull" Jul 10 02:15:25.727949 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:25.727888 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6ea928e-6166-48a9-873d-f5de8a517d75" containerName="pull" Jul 10 02:15:25.727949 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:25.727907 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6ea928e-6166-48a9-873d-f5de8a517d75" containerName="extract" Jul 10 02:15:25.727949 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:25.727912 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6ea928e-6166-48a9-873d-f5de8a517d75" containerName="extract" Jul 10 02:15:25.728151 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:25.727964 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6ea928e-6166-48a9-873d-f5de8a517d75" containerName="extract" Jul 10 02:15:25.729965 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:25.729949 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-p64sc" Jul 10 02:15:25.732881 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:25.732858 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-rz4ds\"" Jul 10 02:15:25.740508 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:25.740488 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-p64sc"] Jul 10 02:15:25.833483 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:25.833411 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8bbb285d-54f6-4d98-aab6-89a6a47fce75-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-p64sc\" (UID: \"8bbb285d-54f6-4d98-aab6-89a6a47fce75\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-p64sc" Jul 10 02:15:25.833483 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:25.833440 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxfmw\" (UniqueName: \"kubernetes.io/projected/8bbb285d-54f6-4d98-aab6-89a6a47fce75-kube-api-access-zxfmw\") pod \"cert-manager-cainjector-559cdd8c87-p64sc\" (UID: \"8bbb285d-54f6-4d98-aab6-89a6a47fce75\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-p64sc" Jul 10 02:15:25.934807 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:25.934770 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8bbb285d-54f6-4d98-aab6-89a6a47fce75-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-p64sc\" (UID: \"8bbb285d-54f6-4d98-aab6-89a6a47fce75\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-p64sc" Jul 10 02:15:25.934807 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:25.934811 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxfmw\" (UniqueName: \"kubernetes.io/projected/8bbb285d-54f6-4d98-aab6-89a6a47fce75-kube-api-access-zxfmw\") pod \"cert-manager-cainjector-559cdd8c87-p64sc\" (UID: \"8bbb285d-54f6-4d98-aab6-89a6a47fce75\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-p64sc" Jul 10 02:15:25.945096 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:25.945067 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8bbb285d-54f6-4d98-aab6-89a6a47fce75-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-p64sc\" (UID: \"8bbb285d-54f6-4d98-aab6-89a6a47fce75\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-p64sc" Jul 10 02:15:25.945352 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:25.945333 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxfmw\" (UniqueName: \"kubernetes.io/projected/8bbb285d-54f6-4d98-aab6-89a6a47fce75-kube-api-access-zxfmw\") pod \"cert-manager-cainjector-559cdd8c87-p64sc\" (UID: \"8bbb285d-54f6-4d98-aab6-89a6a47fce75\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-p64sc" Jul 10 02:15:26.038844 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:26.038799 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-p64sc" Jul 10 02:15:26.166761 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:26.166734 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-p64sc"] Jul 10 02:15:26.168864 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:15:26.168841 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8bbb285d_54f6_4d98_aab6_89a6a47fce75.slice/crio-cdc40d134dd56d8470e9e8da828cae8160b7e7a42b9f3c635aa967a5d95fe8d1 WatchSource:0}: Error finding container cdc40d134dd56d8470e9e8da828cae8160b7e7a42b9f3c635aa967a5d95fe8d1: Status 404 returned error can't find the container with id cdc40d134dd56d8470e9e8da828cae8160b7e7a42b9f3c635aa967a5d95fe8d1 Jul 10 02:15:26.578578 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:26.578547 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-p64sc" event={"ID":"8bbb285d-54f6-4d98-aab6-89a6a47fce75","Type":"ContainerStarted","Data":"cdc40d134dd56d8470e9e8da828cae8160b7e7a42b9f3c635aa967a5d95fe8d1"} Jul 10 02:15:27.227372 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:27.227343 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:15:27.325586 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:27.325546 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:15:29.368327 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:29.368290 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:15:29.594657 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:29.594617 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-p64sc" event={"ID":"8bbb285d-54f6-4d98-aab6-89a6a47fce75","Type":"ContainerStarted","Data":"0c820624e733bf724a20022672db238718beec5c8b5def49f04c40c5c4747365"} Jul 10 02:15:29.611631 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:29.611564 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-p64sc" podStartSLOduration=1.5977160910000001 podStartE2EDuration="4.611550962s" totalImagesPullingTime="3.013834871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:15:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:15:29.610885652 +0000 UTC m=+461.187162739" watchObservedRunningTime="2026-07-10 02:15:29.611550962 +0000 UTC m=+461.187828047" Jul 10 02:15:42.059502 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:42.059422 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:15:42.162655 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:42.162616 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:15:42.649128 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:42.649095 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-xlznj" event={"ID":"254106ce-ff9e-4fdf-b528-6ab449733f07","Type":"ContainerStarted","Data":"b3c2795a42d65803644c7c49999ca0dfe307d5f8bb7433e23a6f00fc60457c57"} Jul 10 02:15:42.649361 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:42.649144 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-xlznj" Jul 10 02:15:42.668414 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:42.668357 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-xlznj" podStartSLOduration=2.273709622 podStartE2EDuration="21.668337782s" totalImagesPullingTime="19.39462816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:15:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:15:42.666688782 +0000 UTC m=+474.242965881" watchObservedRunningTime="2026-07-10 02:15:42.668337782 +0000 UTC m=+474.244614870" Jul 10 02:15:48.655467 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:48.655433 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-xlznj" Jul 10 02:15:48.691401 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:48.691365 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:15:48.794990 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:48.794952 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:15:50.693041 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:50.693012 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbjlqn"] Jul 10 02:15:50.696059 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:50.696043 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbjlqn" Jul 10 02:15:50.700077 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:50.700060 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-mt7hs\"" Jul 10 02:15:50.700176 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:50.700146 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 10 02:15:50.701543 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:50.701523 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 10 02:15:50.704406 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:50.704381 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbjlqn"] Jul 10 02:15:50.823139 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:50.823101 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8wt7\" (UniqueName: \"kubernetes.io/projected/efb3bf53-cc36-4253-a3fe-98a50c22f626-kube-api-access-q8wt7\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbjlqn\" (UID: \"efb3bf53-cc36-4253-a3fe-98a50c22f626\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbjlqn" Jul 10 02:15:50.823139 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:50.823136 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/efb3bf53-cc36-4253-a3fe-98a50c22f626-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbjlqn\" (UID: \"efb3bf53-cc36-4253-a3fe-98a50c22f626\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbjlqn" Jul 10 02:15:50.823325 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:50.823208 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/efb3bf53-cc36-4253-a3fe-98a50c22f626-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbjlqn\" (UID: \"efb3bf53-cc36-4253-a3fe-98a50c22f626\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbjlqn" Jul 10 02:15:50.828319 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:50.828291 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:15:50.924499 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:50.924470 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8wt7\" (UniqueName: \"kubernetes.io/projected/efb3bf53-cc36-4253-a3fe-98a50c22f626-kube-api-access-q8wt7\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbjlqn\" (UID: \"efb3bf53-cc36-4253-a3fe-98a50c22f626\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbjlqn" Jul 10 02:15:50.924690 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:50.924504 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/efb3bf53-cc36-4253-a3fe-98a50c22f626-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbjlqn\" (UID: \"efb3bf53-cc36-4253-a3fe-98a50c22f626\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbjlqn" Jul 10 02:15:50.924690 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:50.924560 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/efb3bf53-cc36-4253-a3fe-98a50c22f626-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbjlqn\" (UID: \"efb3bf53-cc36-4253-a3fe-98a50c22f626\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbjlqn" Jul 10 02:15:50.925014 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:50.924988 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/efb3bf53-cc36-4253-a3fe-98a50c22f626-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbjlqn\" (UID: \"efb3bf53-cc36-4253-a3fe-98a50c22f626\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbjlqn" Jul 10 02:15:50.925136 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:50.925000 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/efb3bf53-cc36-4253-a3fe-98a50c22f626-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbjlqn\" (UID: \"efb3bf53-cc36-4253-a3fe-98a50c22f626\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbjlqn" Jul 10 02:15:50.937675 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:50.937651 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8wt7\" (UniqueName: \"kubernetes.io/projected/efb3bf53-cc36-4253-a3fe-98a50c22f626-kube-api-access-q8wt7\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbjlqn\" (UID: \"efb3bf53-cc36-4253-a3fe-98a50c22f626\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbjlqn" Jul 10 02:15:51.005289 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:51.005225 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbjlqn" Jul 10 02:15:51.135510 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:51.135473 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbjlqn"] Jul 10 02:15:51.138758 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:15:51.138734 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefb3bf53_cc36_4253_a3fe_98a50c22f626.slice/crio-d5238ce32b124152a50b062f3ae5fc89fa56b4dd2fcef316d246b09147cf0914 WatchSource:0}: Error finding container d5238ce32b124152a50b062f3ae5fc89fa56b4dd2fcef316d246b09147cf0914: Status 404 returned error can't find the container with id d5238ce32b124152a50b062f3ae5fc89fa56b4dd2fcef316d246b09147cf0914 Jul 10 02:15:51.697366 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:51.697327 2566 generic.go:356] "Generic (PLEG): container finished" podID="efb3bf53-cc36-4253-a3fe-98a50c22f626" containerID="e806f8fbdb723c3babc81d0d35ecc1a81fd4a98818e9b2e7424b8882761b58f1" exitCode=0 Jul 10 02:15:51.697863 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:51.697415 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbjlqn" event={"ID":"efb3bf53-cc36-4253-a3fe-98a50c22f626","Type":"ContainerDied","Data":"e806f8fbdb723c3babc81d0d35ecc1a81fd4a98818e9b2e7424b8882761b58f1"} Jul 10 02:15:51.697863 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:51.697492 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbjlqn" event={"ID":"efb3bf53-cc36-4253-a3fe-98a50c22f626","Type":"ContainerStarted","Data":"d5238ce32b124152a50b062f3ae5fc89fa56b4dd2fcef316d246b09147cf0914"} Jul 10 02:15:52.703407 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:52.703374 2566 generic.go:356] "Generic (PLEG): container finished" podID="efb3bf53-cc36-4253-a3fe-98a50c22f626" containerID="552ff96cc29acce5580004d6d6668cbe18372301af20b885e4df3bbb43292665" exitCode=0 Jul 10 02:15:52.703790 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:52.703425 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbjlqn" event={"ID":"efb3bf53-cc36-4253-a3fe-98a50c22f626","Type":"ContainerDied","Data":"552ff96cc29acce5580004d6d6668cbe18372301af20b885e4df3bbb43292665"} Jul 10 02:15:53.709894 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:53.709860 2566 generic.go:356] "Generic (PLEG): container finished" podID="efb3bf53-cc36-4253-a3fe-98a50c22f626" containerID="9f4a6e4b70a0bfea8e9177b94a32fa1232ea4f6e851c74bb49f97d8bf34d88a0" exitCode=0 Jul 10 02:15:53.710274 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:53.709937 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbjlqn" event={"ID":"efb3bf53-cc36-4253-a3fe-98a50c22f626","Type":"ContainerDied","Data":"9f4a6e4b70a0bfea8e9177b94a32fa1232ea4f6e851c74bb49f97d8bf34d88a0"} Jul 10 02:15:54.841451 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:54.841428 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbjlqn" Jul 10 02:15:54.956843 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:54.956817 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/efb3bf53-cc36-4253-a3fe-98a50c22f626-bundle\") pod \"efb3bf53-cc36-4253-a3fe-98a50c22f626\" (UID: \"efb3bf53-cc36-4253-a3fe-98a50c22f626\") " Jul 10 02:15:54.957004 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:54.956868 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/efb3bf53-cc36-4253-a3fe-98a50c22f626-util\") pod \"efb3bf53-cc36-4253-a3fe-98a50c22f626\" (UID: \"efb3bf53-cc36-4253-a3fe-98a50c22f626\") " Jul 10 02:15:54.957004 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:54.956896 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8wt7\" (UniqueName: \"kubernetes.io/projected/efb3bf53-cc36-4253-a3fe-98a50c22f626-kube-api-access-q8wt7\") pod \"efb3bf53-cc36-4253-a3fe-98a50c22f626\" (UID: \"efb3bf53-cc36-4253-a3fe-98a50c22f626\") " Jul 10 02:15:54.957482 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:54.957457 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb3bf53-cc36-4253-a3fe-98a50c22f626-bundle" (OuterVolumeSpecName: "bundle") pod "efb3bf53-cc36-4253-a3fe-98a50c22f626" (UID: "efb3bf53-cc36-4253-a3fe-98a50c22f626"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 02:15:54.959147 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:54.959120 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efb3bf53-cc36-4253-a3fe-98a50c22f626-kube-api-access-q8wt7" (OuterVolumeSpecName: "kube-api-access-q8wt7") pod "efb3bf53-cc36-4253-a3fe-98a50c22f626" (UID: "efb3bf53-cc36-4253-a3fe-98a50c22f626"). InnerVolumeSpecName "kube-api-access-q8wt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 02:15:54.963019 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:54.962997 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb3bf53-cc36-4253-a3fe-98a50c22f626-util" (OuterVolumeSpecName: "util") pod "efb3bf53-cc36-4253-a3fe-98a50c22f626" (UID: "efb3bf53-cc36-4253-a3fe-98a50c22f626"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 02:15:55.031083 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:55.031054 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:15:55.057837 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:55.057806 2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/efb3bf53-cc36-4253-a3fe-98a50c22f626-util\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:15:55.057837 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:55.057832 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8wt7\" (UniqueName: \"kubernetes.io/projected/efb3bf53-cc36-4253-a3fe-98a50c22f626-kube-api-access-q8wt7\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:15:55.057996 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:55.057846 2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/efb3bf53-cc36-4253-a3fe-98a50c22f626-bundle\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:15:55.099630 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:55.099584 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-58d7cbdf66-2v8tx"] Jul 10 02:15:55.099968 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:55.099951 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efb3bf53-cc36-4253-a3fe-98a50c22f626" containerName="util" Jul 10 02:15:55.100049 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:55.099971 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="efb3bf53-cc36-4253-a3fe-98a50c22f626" containerName="util" Jul 10 02:15:55.100049 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:55.099994 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efb3bf53-cc36-4253-a3fe-98a50c22f626" containerName="extract" Jul 10 02:15:55.100049 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:55.100003 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="efb3bf53-cc36-4253-a3fe-98a50c22f626" containerName="extract" Jul 10 02:15:55.100049 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:55.100016 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efb3bf53-cc36-4253-a3fe-98a50c22f626" containerName="pull" Jul 10 02:15:55.100049 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:55.100024 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="efb3bf53-cc36-4253-a3fe-98a50c22f626" containerName="pull" Jul 10 02:15:55.100275 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:55.100112 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="efb3bf53-cc36-4253-a3fe-98a50c22f626" containerName="extract" Jul 10 02:15:55.102078 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:55.102058 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-58d7cbdf66-2v8tx" Jul 10 02:15:55.104951 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:55.104876 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 10 02:15:55.105150 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:55.105127 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 10 02:15:55.105255 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:55.105164 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 10 02:15:55.105255 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:55.105232 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 10 02:15:55.105364 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:55.105279 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 10 02:15:55.105466 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:55.105448 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-x9c97\"" Jul 10 02:15:55.106400 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:55.106375 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:15:55.111751 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:55.111733 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-58d7cbdf66-2v8tx"] Jul 10 02:15:55.159061 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:55.159011 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4ff2f2ed-7d54-46f5-98ab-0a99f29ffe09-manager-config\") pod \"lws-controller-manager-58d7cbdf66-2v8tx\" (UID: \"4ff2f2ed-7d54-46f5-98ab-0a99f29ffe09\") " pod="openshift-lws-operator/lws-controller-manager-58d7cbdf66-2v8tx" Jul 10 02:15:55.159216 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:55.159112 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4ff2f2ed-7d54-46f5-98ab-0a99f29ffe09-metrics-cert\") pod \"lws-controller-manager-58d7cbdf66-2v8tx\" (UID: \"4ff2f2ed-7d54-46f5-98ab-0a99f29ffe09\") " pod="openshift-lws-operator/lws-controller-manager-58d7cbdf66-2v8tx" Jul 10 02:15:55.159216 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:55.159185 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ff2f2ed-7d54-46f5-98ab-0a99f29ffe09-cert\") pod \"lws-controller-manager-58d7cbdf66-2v8tx\" (UID: \"4ff2f2ed-7d54-46f5-98ab-0a99f29ffe09\") " pod="openshift-lws-operator/lws-controller-manager-58d7cbdf66-2v8tx" Jul 10 02:15:55.159331 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:55.159219 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjswt\" (UniqueName: \"kubernetes.io/projected/4ff2f2ed-7d54-46f5-98ab-0a99f29ffe09-kube-api-access-hjswt\") pod \"lws-controller-manager-58d7cbdf66-2v8tx\" (UID: \"4ff2f2ed-7d54-46f5-98ab-0a99f29ffe09\") " pod="openshift-lws-operator/lws-controller-manager-58d7cbdf66-2v8tx" Jul 10 02:15:55.260065 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:55.260030 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4ff2f2ed-7d54-46f5-98ab-0a99f29ffe09-metrics-cert\") pod \"lws-controller-manager-58d7cbdf66-2v8tx\" (UID: \"4ff2f2ed-7d54-46f5-98ab-0a99f29ffe09\") " pod="openshift-lws-operator/lws-controller-manager-58d7cbdf66-2v8tx" Jul 10 02:15:55.260194 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:55.260111 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ff2f2ed-7d54-46f5-98ab-0a99f29ffe09-cert\") pod \"lws-controller-manager-58d7cbdf66-2v8tx\" (UID: \"4ff2f2ed-7d54-46f5-98ab-0a99f29ffe09\") " pod="openshift-lws-operator/lws-controller-manager-58d7cbdf66-2v8tx" Jul 10 02:15:55.260194 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:55.260154 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjswt\" (UniqueName: \"kubernetes.io/projected/4ff2f2ed-7d54-46f5-98ab-0a99f29ffe09-kube-api-access-hjswt\") pod \"lws-controller-manager-58d7cbdf66-2v8tx\" (UID: \"4ff2f2ed-7d54-46f5-98ab-0a99f29ffe09\") " pod="openshift-lws-operator/lws-controller-manager-58d7cbdf66-2v8tx" Jul 10 02:15:55.260354 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:55.260335 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4ff2f2ed-7d54-46f5-98ab-0a99f29ffe09-manager-config\") pod \"lws-controller-manager-58d7cbdf66-2v8tx\" (UID: \"4ff2f2ed-7d54-46f5-98ab-0a99f29ffe09\") " pod="openshift-lws-operator/lws-controller-manager-58d7cbdf66-2v8tx" Jul 10 02:15:55.261065 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:55.261039 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4ff2f2ed-7d54-46f5-98ab-0a99f29ffe09-manager-config\") pod \"lws-controller-manager-58d7cbdf66-2v8tx\" (UID: \"4ff2f2ed-7d54-46f5-98ab-0a99f29ffe09\") " pod="openshift-lws-operator/lws-controller-manager-58d7cbdf66-2v8tx" Jul 10 02:15:55.262320 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:55.262298 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4ff2f2ed-7d54-46f5-98ab-0a99f29ffe09-metrics-cert\") pod \"lws-controller-manager-58d7cbdf66-2v8tx\" (UID: \"4ff2f2ed-7d54-46f5-98ab-0a99f29ffe09\") " pod="openshift-lws-operator/lws-controller-manager-58d7cbdf66-2v8tx" Jul 10 02:15:55.262431 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:55.262416 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ff2f2ed-7d54-46f5-98ab-0a99f29ffe09-cert\") pod \"lws-controller-manager-58d7cbdf66-2v8tx\" (UID: \"4ff2f2ed-7d54-46f5-98ab-0a99f29ffe09\") " pod="openshift-lws-operator/lws-controller-manager-58d7cbdf66-2v8tx" Jul 10 02:15:55.271019 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:55.270967 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjswt\" (UniqueName: \"kubernetes.io/projected/4ff2f2ed-7d54-46f5-98ab-0a99f29ffe09-kube-api-access-hjswt\") pod \"lws-controller-manager-58d7cbdf66-2v8tx\" (UID: \"4ff2f2ed-7d54-46f5-98ab-0a99f29ffe09\") " pod="openshift-lws-operator/lws-controller-manager-58d7cbdf66-2v8tx" Jul 10 02:15:55.411851 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:55.411822 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-58d7cbdf66-2v8tx" Jul 10 02:15:55.568424 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:55.568307 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-58d7cbdf66-2v8tx"] Jul 10 02:15:55.570940 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:15:55.570916 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ff2f2ed_7d54_46f5_98ab_0a99f29ffe09.slice/crio-e1b42ccc0d3fb3edef964b219fec67ffd96397faa119ea0ea59f668e28a0d6fe WatchSource:0}: Error finding container e1b42ccc0d3fb3edef964b219fec67ffd96397faa119ea0ea59f668e28a0d6fe: Status 404 returned error can't find the container with id e1b42ccc0d3fb3edef964b219fec67ffd96397faa119ea0ea59f668e28a0d6fe Jul 10 02:15:55.719586 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:55.719552 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-58d7cbdf66-2v8tx" event={"ID":"4ff2f2ed-7d54-46f5-98ab-0a99f29ffe09","Type":"ContainerStarted","Data":"e1b42ccc0d3fb3edef964b219fec67ffd96397faa119ea0ea59f668e28a0d6fe"} Jul 10 02:15:55.721306 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:55.721290 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbjlqn" Jul 10 02:15:55.721306 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:55.721294 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbjlqn" event={"ID":"efb3bf53-cc36-4253-a3fe-98a50c22f626","Type":"ContainerDied","Data":"d5238ce32b124152a50b062f3ae5fc89fa56b4dd2fcef316d246b09147cf0914"} Jul 10 02:15:55.721405 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:55.721327 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5238ce32b124152a50b062f3ae5fc89fa56b4dd2fcef316d246b09147cf0914" Jul 10 02:15:57.733403 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:57.733368 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-58d7cbdf66-2v8tx" event={"ID":"4ff2f2ed-7d54-46f5-98ab-0a99f29ffe09","Type":"ContainerStarted","Data":"a95a46ed74de24fe19d5b0d444139ed9c3c094dd4e6ce272f84934a7a39a3932"} Jul 10 02:15:57.733801 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:57.733496 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-58d7cbdf66-2v8tx" Jul 10 02:15:57.752997 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:15:57.752952 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-58d7cbdf66-2v8tx" podStartSLOduration=1.127636279 podStartE2EDuration="2.752936079s" totalImagesPullingTime="1.6252998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:15:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:15:57.750943641 +0000 UTC m=+489.327220728" watchObservedRunningTime="2026-07-10 02:15:57.752936079 +0000 UTC m=+489.329213164" Jul 10 02:16:08.019377 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:08.019340 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:16:08.128434 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:08.128391 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:16:08.740731 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:08.740703 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-58d7cbdf66-2v8tx" Jul 10 02:16:08.826194 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:08.826161 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:16:09.025929 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:09.025894 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:16:09.130439 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:09.130408 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:16:09.213194 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:09.213163 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-78b9cbb96-rl82t"] Jul 10 02:16:09.216274 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:09.216251 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-78b9cbb96-rl82t" Jul 10 02:16:09.219025 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:09.218999 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 10 02:16:09.219186 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:09.219127 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 10 02:16:09.219268 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:09.219245 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 10 02:16:09.219319 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:09.219272 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-rnfzq\"" Jul 10 02:16:09.219358 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:09.219330 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 10 02:16:09.234557 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:09.234532 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-78b9cbb96-rl82t"] Jul 10 02:16:09.343841 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:09.343753 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:16:09.378807 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:09.378769 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d85d3688-f7b6-4739-a498-33484bd9b45f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-78b9cbb96-rl82t\" (UID: \"d85d3688-f7b6-4739-a498-33484bd9b45f\") " pod="opendatahub/opendatahub-operator-controller-manager-78b9cbb96-rl82t" Jul 10 02:16:09.378990 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:09.378812 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d85d3688-f7b6-4739-a498-33484bd9b45f-webhook-cert\") pod \"opendatahub-operator-controller-manager-78b9cbb96-rl82t\" (UID: \"d85d3688-f7b6-4739-a498-33484bd9b45f\") " pod="opendatahub/opendatahub-operator-controller-manager-78b9cbb96-rl82t" Jul 10 02:16:09.378990 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:09.378847 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxszg\" (UniqueName: \"kubernetes.io/projected/d85d3688-f7b6-4739-a498-33484bd9b45f-kube-api-access-cxszg\") pod \"opendatahub-operator-controller-manager-78b9cbb96-rl82t\" (UID: \"d85d3688-f7b6-4739-a498-33484bd9b45f\") " pod="opendatahub/opendatahub-operator-controller-manager-78b9cbb96-rl82t" Jul 10 02:16:09.378990 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:09.378915 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d85d3688-f7b6-4739-a498-33484bd9b45f-manifests\") pod \"opendatahub-operator-controller-manager-78b9cbb96-rl82t\" (UID: \"d85d3688-f7b6-4739-a498-33484bd9b45f\") " pod="opendatahub/opendatahub-operator-controller-manager-78b9cbb96-rl82t" Jul 10 02:16:09.378990 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:09.378942 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d85d3688-f7b6-4739-a498-33484bd9b45f-tmp\") pod \"opendatahub-operator-controller-manager-78b9cbb96-rl82t\" (UID: \"d85d3688-f7b6-4739-a498-33484bd9b45f\") " pod="opendatahub/opendatahub-operator-controller-manager-78b9cbb96-rl82t" Jul 10 02:16:09.436579 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:09.436534 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:16:09.479689 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:09.479654 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d85d3688-f7b6-4739-a498-33484bd9b45f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-78b9cbb96-rl82t\" (UID: \"d85d3688-f7b6-4739-a498-33484bd9b45f\") " pod="opendatahub/opendatahub-operator-controller-manager-78b9cbb96-rl82t" Jul 10 02:16:09.479689 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:09.479691 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d85d3688-f7b6-4739-a498-33484bd9b45f-webhook-cert\") pod \"opendatahub-operator-controller-manager-78b9cbb96-rl82t\" (UID: \"d85d3688-f7b6-4739-a498-33484bd9b45f\") " pod="opendatahub/opendatahub-operator-controller-manager-78b9cbb96-rl82t" Jul 10 02:16:09.479922 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:09.479711 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxszg\" (UniqueName: \"kubernetes.io/projected/d85d3688-f7b6-4739-a498-33484bd9b45f-kube-api-access-cxszg\") pod \"opendatahub-operator-controller-manager-78b9cbb96-rl82t\" (UID: \"d85d3688-f7b6-4739-a498-33484bd9b45f\") " pod="opendatahub/opendatahub-operator-controller-manager-78b9cbb96-rl82t" Jul 10 02:16:09.479922 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:09.479758 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d85d3688-f7b6-4739-a498-33484bd9b45f-manifests\") pod \"opendatahub-operator-controller-manager-78b9cbb96-rl82t\" (UID: \"d85d3688-f7b6-4739-a498-33484bd9b45f\") " pod="opendatahub/opendatahub-operator-controller-manager-78b9cbb96-rl82t" Jul 10 02:16:09.479922 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:09.479779 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d85d3688-f7b6-4739-a498-33484bd9b45f-tmp\") pod \"opendatahub-operator-controller-manager-78b9cbb96-rl82t\" (UID: \"d85d3688-f7b6-4739-a498-33484bd9b45f\") " pod="opendatahub/opendatahub-operator-controller-manager-78b9cbb96-rl82t" Jul 10 02:16:09.480197 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:09.480166 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d85d3688-f7b6-4739-a498-33484bd9b45f-manifests\") pod \"opendatahub-operator-controller-manager-78b9cbb96-rl82t\" (UID: \"d85d3688-f7b6-4739-a498-33484bd9b45f\") " pod="opendatahub/opendatahub-operator-controller-manager-78b9cbb96-rl82t" Jul 10 02:16:09.482047 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:09.482024 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d85d3688-f7b6-4739-a498-33484bd9b45f-tmp\") pod \"opendatahub-operator-controller-manager-78b9cbb96-rl82t\" (UID: \"d85d3688-f7b6-4739-a498-33484bd9b45f\") " pod="opendatahub/opendatahub-operator-controller-manager-78b9cbb96-rl82t" Jul 10 02:16:09.482237 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:09.482220 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d85d3688-f7b6-4739-a498-33484bd9b45f-webhook-cert\") pod \"opendatahub-operator-controller-manager-78b9cbb96-rl82t\" (UID: \"d85d3688-f7b6-4739-a498-33484bd9b45f\") " pod="opendatahub/opendatahub-operator-controller-manager-78b9cbb96-rl82t" Jul 10 02:16:09.482299 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:09.482260 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d85d3688-f7b6-4739-a498-33484bd9b45f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-78b9cbb96-rl82t\" (UID: \"d85d3688-f7b6-4739-a498-33484bd9b45f\") " pod="opendatahub/opendatahub-operator-controller-manager-78b9cbb96-rl82t" Jul 10 02:16:09.489187 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:09.489166 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxszg\" (UniqueName: \"kubernetes.io/projected/d85d3688-f7b6-4739-a498-33484bd9b45f-kube-api-access-cxszg\") pod \"opendatahub-operator-controller-manager-78b9cbb96-rl82t\" (UID: \"d85d3688-f7b6-4739-a498-33484bd9b45f\") " pod="opendatahub/opendatahub-operator-controller-manager-78b9cbb96-rl82t" Jul 10 02:16:09.497647 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:09.497621 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:16:09.526650 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:09.526626 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-78b9cbb96-rl82t" Jul 10 02:16:09.600712 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:09.600633 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:16:09.669509 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:09.669479 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-78b9cbb96-rl82t"] Jul 10 02:16:09.674029 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:16:09.673998 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd85d3688_f7b6_4739_a498_33484bd9b45f.slice/crio-6c3e61662e0b94fc3b597dee499cf7a88749559ed531093e7a44b7c3e133c22e WatchSource:0}: Error finding container 6c3e61662e0b94fc3b597dee499cf7a88749559ed531093e7a44b7c3e133c22e: Status 404 returned error can't find the container with id 6c3e61662e0b94fc3b597dee499cf7a88749559ed531093e7a44b7c3e133c22e Jul 10 02:16:09.698891 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:09.698861 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:16:09.786995 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:09.786964 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-78b9cbb96-rl82t" event={"ID":"d85d3688-f7b6-4739-a498-33484bd9b45f","Type":"ContainerStarted","Data":"6c3e61662e0b94fc3b597dee499cf7a88749559ed531093e7a44b7c3e133c22e"} Jul 10 02:16:12.805751 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:12.805712 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-78b9cbb96-rl82t" event={"ID":"d85d3688-f7b6-4739-a498-33484bd9b45f","Type":"ContainerStarted","Data":"e0d3363eeefac980c0328ac4be47fc97b5ccaf2b57efed0c60dc254704962cc1"} Jul 10 02:16:13.811371 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:13.811334 2566 generic.go:356] "Generic (PLEG): container finished" podID="d85d3688-f7b6-4739-a498-33484bd9b45f" containerID="e0d3363eeefac980c0328ac4be47fc97b5ccaf2b57efed0c60dc254704962cc1" exitCode=0 Jul 10 02:16:13.811871 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:13.811416 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-78b9cbb96-rl82t" event={"ID":"d85d3688-f7b6-4739-a498-33484bd9b45f","Type":"ContainerDied","Data":"e0d3363eeefac980c0328ac4be47fc97b5ccaf2b57efed0c60dc254704962cc1"} Jul 10 02:16:14.817939 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:14.817902 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-78b9cbb96-rl82t" event={"ID":"d85d3688-f7b6-4739-a498-33484bd9b45f","Type":"ContainerStarted","Data":"87abf870207b3aafb486686ddcbacce3a23dae7b06247e5c5fbd36dd84ce7920"} Jul 10 02:16:14.818363 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:14.818024 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-78b9cbb96-rl82t" Jul 10 02:16:14.845405 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:14.845354 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-78b9cbb96-rl82t" podStartSLOduration=1.563341839 podStartE2EDuration="5.845338251s" totalImagesPullingTime="3.280576645s" totalInitContainerRuntime="1.001419767s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:16:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:16:14.842661487 +0000 UTC m=+506.418938591" watchObservedRunningTime="2026-07-10 02:16:14.845338251 +0000 UTC m=+506.421615338" Jul 10 02:16:21.215881 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:21.215839 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:16:21.330358 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:21.330323 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:16:25.824412 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:25.824383 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-78b9cbb96-rl82t" Jul 10 02:16:28.131520 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:28.131480 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:16:28.271527 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:28.271493 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:16:28.616670 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:28.616632 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:16:28.999455 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:28.999366 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:16:29.132943 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:29.132889 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:16:29.595702 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:29.595643 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:16:29.901936 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:29.901832 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:16:30.036814 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:30.036778 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:16:30.129576 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:30.129538 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:16:31.034937 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:31.034890 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:16:35.973478 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:35.973447 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9ts97"] Jul 10 02:16:35.983218 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:35.983188 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9ts97" Jul 10 02:16:35.986423 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:35.986401 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 10 02:16:35.986587 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:35.986413 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 10 02:16:35.986731 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:35.986431 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 10 02:16:35.987768 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:35.987741 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 10 02:16:35.987872 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:35.987769 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-vj8ps\"" Jul 10 02:16:35.988754 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:35.988735 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9ts97"] Jul 10 02:16:35.991782 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:35.991758 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz8hv\" (UniqueName: \"kubernetes.io/projected/49d07846-4ea0-496e-843b-90a8920ea7bc-kube-api-access-vz8hv\") pod \"istiod-openshift-gateway-94bb8fbfd-9ts97\" (UID: \"49d07846-4ea0-496e-843b-90a8920ea7bc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9ts97" Jul 10 02:16:35.991887 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:35.991787 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/49d07846-4ea0-496e-843b-90a8920ea7bc-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-9ts97\" (UID: \"49d07846-4ea0-496e-843b-90a8920ea7bc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9ts97" Jul 10 02:16:35.991887 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:35.991826 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/49d07846-4ea0-496e-843b-90a8920ea7bc-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-9ts97\" (UID: \"49d07846-4ea0-496e-843b-90a8920ea7bc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9ts97" Jul 10 02:16:35.991887 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:35.991842 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/49d07846-4ea0-496e-843b-90a8920ea7bc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-9ts97\" (UID: \"49d07846-4ea0-496e-843b-90a8920ea7bc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9ts97" Jul 10 02:16:35.992049 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:35.991942 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49d07846-4ea0-496e-843b-90a8920ea7bc-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-9ts97\" (UID: \"49d07846-4ea0-496e-843b-90a8920ea7bc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9ts97" Jul 10 02:16:35.992049 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:35.991967 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/49d07846-4ea0-496e-843b-90a8920ea7bc-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-9ts97\" (UID: \"49d07846-4ea0-496e-843b-90a8920ea7bc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9ts97" Jul 10 02:16:35.992049 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:35.991986 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/49d07846-4ea0-496e-843b-90a8920ea7bc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-9ts97\" (UID: \"49d07846-4ea0-496e-843b-90a8920ea7bc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9ts97" Jul 10 02:16:36.092966 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:36.092934 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49d07846-4ea0-496e-843b-90a8920ea7bc-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-9ts97\" (UID: \"49d07846-4ea0-496e-843b-90a8920ea7bc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9ts97" Jul 10 02:16:36.093177 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:36.092979 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/49d07846-4ea0-496e-843b-90a8920ea7bc-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-9ts97\" (UID: \"49d07846-4ea0-496e-843b-90a8920ea7bc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9ts97" Jul 10 02:16:36.093177 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:36.093011 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/49d07846-4ea0-496e-843b-90a8920ea7bc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-9ts97\" (UID: \"49d07846-4ea0-496e-843b-90a8920ea7bc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9ts97" Jul 10 02:16:36.093177 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:36.093042 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz8hv\" (UniqueName: \"kubernetes.io/projected/49d07846-4ea0-496e-843b-90a8920ea7bc-kube-api-access-vz8hv\") pod \"istiod-openshift-gateway-94bb8fbfd-9ts97\" (UID: \"49d07846-4ea0-496e-843b-90a8920ea7bc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9ts97" Jul 10 02:16:36.093177 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:36.093066 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/49d07846-4ea0-496e-843b-90a8920ea7bc-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-9ts97\" (UID: \"49d07846-4ea0-496e-843b-90a8920ea7bc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9ts97" Jul 10 02:16:36.093177 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:36.093106 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/49d07846-4ea0-496e-843b-90a8920ea7bc-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-9ts97\" (UID: \"49d07846-4ea0-496e-843b-90a8920ea7bc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9ts97" Jul 10 02:16:36.093403 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:36.093220 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/49d07846-4ea0-496e-843b-90a8920ea7bc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-9ts97\" (UID: \"49d07846-4ea0-496e-843b-90a8920ea7bc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9ts97" Jul 10 02:16:36.093680 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:36.093653 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/49d07846-4ea0-496e-843b-90a8920ea7bc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-9ts97\" (UID: \"49d07846-4ea0-496e-843b-90a8920ea7bc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9ts97" Jul 10 02:16:36.095526 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:36.095503 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/49d07846-4ea0-496e-843b-90a8920ea7bc-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-9ts97\" (UID: \"49d07846-4ea0-496e-843b-90a8920ea7bc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9ts97" Jul 10 02:16:36.095654 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:36.095566 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/49d07846-4ea0-496e-843b-90a8920ea7bc-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-9ts97\" (UID: \"49d07846-4ea0-496e-843b-90a8920ea7bc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9ts97" Jul 10 02:16:36.095654 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:36.095578 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/49d07846-4ea0-496e-843b-90a8920ea7bc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-9ts97\" (UID: \"49d07846-4ea0-496e-843b-90a8920ea7bc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9ts97" Jul 10 02:16:36.095733 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:36.095673 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/49d07846-4ea0-496e-843b-90a8920ea7bc-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-9ts97\" (UID: \"49d07846-4ea0-496e-843b-90a8920ea7bc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9ts97" Jul 10 02:16:36.102345 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:36.102313 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49d07846-4ea0-496e-843b-90a8920ea7bc-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-9ts97\" (UID: \"49d07846-4ea0-496e-843b-90a8920ea7bc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9ts97" Jul 10 02:16:36.102936 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:36.102917 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz8hv\" (UniqueName: \"kubernetes.io/projected/49d07846-4ea0-496e-843b-90a8920ea7bc-kube-api-access-vz8hv\") pod \"istiod-openshift-gateway-94bb8fbfd-9ts97\" (UID: \"49d07846-4ea0-496e-843b-90a8920ea7bc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9ts97" Jul 10 02:16:36.294170 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:36.294101 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9ts97" Jul 10 02:16:36.338565 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:36.338533 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:16:36.429759 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:36.429728 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9ts97"] Jul 10 02:16:36.431795 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:16:36.431767 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49d07846_4ea0_496e_843b_90a8920ea7bc.slice/crio-fc6a729493c399103d92d3daf0c93d958bd2ad440ab041130f8b07afc19072cd WatchSource:0}: Error finding container fc6a729493c399103d92d3daf0c93d958bd2ad440ab041130f8b07afc19072cd: Status 404 returned error can't find the container with id fc6a729493c399103d92d3daf0c93d958bd2ad440ab041130f8b07afc19072cd Jul 10 02:16:36.433803 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:36.433651 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:16:36.927238 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:36.927204 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9ts97" event={"ID":"49d07846-4ea0-496e-843b-90a8920ea7bc","Type":"ContainerStarted","Data":"fc6a729493c399103d92d3daf0c93d958bd2ad440ab041130f8b07afc19072cd"} Jul 10 02:16:37.131353 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:37.131314 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:16:37.443139 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:37.443096 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:16:37.543109 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:37.542935 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:16:38.946361 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:38.946322 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:16:39.041928 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:39.041844 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:16:39.192041 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:39.191987 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:16:39.292800 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:39.292760 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 02:16:39.292927 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:39.292844 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 02:16:39.948199 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:39.948158 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9ts97" event={"ID":"49d07846-4ea0-496e-843b-90a8920ea7bc","Type":"ContainerStarted","Data":"e64d1a6f0f0b4c233cb7a7fb139937e655a0b27f5a725015c12ae7a3f58d5d8f"} Jul 10 02:16:39.948582 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:39.948297 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9ts97" Jul 10 02:16:39.984418 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:39.984357 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9ts97" podStartSLOduration=2.126072866 podStartE2EDuration="4.984337564s" totalImagesPullingTime="2.858264698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:16:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:16:39.980926037 +0000 UTC m=+531.557203125" watchObservedRunningTime="2026-07-10 02:16:39.984337564 +0000 UTC m=+531.560614651" Jul 10 02:16:40.088806 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:40.088639 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:16:40.235852 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:40.235777 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:16:40.337508 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:40.337454 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:16:40.450260 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:40.450174 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:16:40.969978 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:16:40.969944 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9ts97" Jul 10 02:17:01.322341 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:17:01.322307 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:17:03.275052 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:17:03.275020 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:17:03.336895 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:17:03.336864 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:17:48.876549 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:17:48.876523 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-686777b996-krg22"] Jul 10 02:17:48.878819 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:17:48.878800 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-686777b996-krg22" Jul 10 02:17:48.891432 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:17:48.891409 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-686777b996-krg22"] Jul 10 02:17:48.943124 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:17:48.943095 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:17:48.971652 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:17:48.971625 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca21095f-2156-4769-851c-c06475e23f85-console-serving-cert\") pod \"console-686777b996-krg22\" (UID: \"ca21095f-2156-4769-851c-c06475e23f85\") " pod="openshift-console/console-686777b996-krg22" Jul 10 02:17:48.971787 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:17:48.971663 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ca21095f-2156-4769-851c-c06475e23f85-console-config\") pod \"console-686777b996-krg22\" (UID: \"ca21095f-2156-4769-851c-c06475e23f85\") " pod="openshift-console/console-686777b996-krg22" Jul 10 02:17:48.971787 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:17:48.971701 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w5j4\" (UniqueName: \"kubernetes.io/projected/ca21095f-2156-4769-851c-c06475e23f85-kube-api-access-5w5j4\") pod \"console-686777b996-krg22\" (UID: \"ca21095f-2156-4769-851c-c06475e23f85\") " pod="openshift-console/console-686777b996-krg22" Jul 10 02:17:48.971787 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:17:48.971739 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ca21095f-2156-4769-851c-c06475e23f85-service-ca\") pod \"console-686777b996-krg22\" (UID: \"ca21095f-2156-4769-851c-c06475e23f85\") " pod="openshift-console/console-686777b996-krg22" Jul 10 02:17:48.971787 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:17:48.971761 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca21095f-2156-4769-851c-c06475e23f85-trusted-ca-bundle\") pod \"console-686777b996-krg22\" (UID: \"ca21095f-2156-4769-851c-c06475e23f85\") " pod="openshift-console/console-686777b996-krg22" Jul 10 02:17:48.971787 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:17:48.971787 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ca21095f-2156-4769-851c-c06475e23f85-console-oauth-config\") pod \"console-686777b996-krg22\" (UID: \"ca21095f-2156-4769-851c-c06475e23f85\") " pod="openshift-console/console-686777b996-krg22" Jul 10 02:17:48.971965 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:17:48.971843 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ca21095f-2156-4769-851c-c06475e23f85-oauth-serving-cert\") pod \"console-686777b996-krg22\" (UID: \"ca21095f-2156-4769-851c-c06475e23f85\") " pod="openshift-console/console-686777b996-krg22" Jul 10 02:17:49.072457 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:17:49.072432 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca21095f-2156-4769-851c-c06475e23f85-console-serving-cert\") pod \"console-686777b996-krg22\" (UID: \"ca21095f-2156-4769-851c-c06475e23f85\") " pod="openshift-console/console-686777b996-krg22" Jul 10 02:17:49.072457 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:17:49.072459 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ca21095f-2156-4769-851c-c06475e23f85-console-config\") pod \"console-686777b996-krg22\" (UID: \"ca21095f-2156-4769-851c-c06475e23f85\") " pod="openshift-console/console-686777b996-krg22" Jul 10 02:17:49.072693 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:17:49.072485 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w5j4\" (UniqueName: \"kubernetes.io/projected/ca21095f-2156-4769-851c-c06475e23f85-kube-api-access-5w5j4\") pod \"console-686777b996-krg22\" (UID: \"ca21095f-2156-4769-851c-c06475e23f85\") " pod="openshift-console/console-686777b996-krg22" Jul 10 02:17:49.072693 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:17:49.072635 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ca21095f-2156-4769-851c-c06475e23f85-service-ca\") pod \"console-686777b996-krg22\" (UID: \"ca21095f-2156-4769-851c-c06475e23f85\") " pod="openshift-console/console-686777b996-krg22" Jul 10 02:17:49.072693 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:17:49.072661 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca21095f-2156-4769-851c-c06475e23f85-trusted-ca-bundle\") pod \"console-686777b996-krg22\" (UID: \"ca21095f-2156-4769-851c-c06475e23f85\") " pod="openshift-console/console-686777b996-krg22" Jul 10 02:17:49.072693 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:17:49.072691 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ca21095f-2156-4769-851c-c06475e23f85-console-oauth-config\") pod \"console-686777b996-krg22\" (UID: \"ca21095f-2156-4769-851c-c06475e23f85\") " pod="openshift-console/console-686777b996-krg22" Jul 10 02:17:49.072902 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:17:49.072756 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ca21095f-2156-4769-851c-c06475e23f85-oauth-serving-cert\") pod \"console-686777b996-krg22\" (UID: \"ca21095f-2156-4769-851c-c06475e23f85\") " pod="openshift-console/console-686777b996-krg22" Jul 10 02:17:49.073219 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:17:49.073190 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ca21095f-2156-4769-851c-c06475e23f85-console-config\") pod \"console-686777b996-krg22\" (UID: \"ca21095f-2156-4769-851c-c06475e23f85\") " pod="openshift-console/console-686777b996-krg22" Jul 10 02:17:49.073348 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:17:49.073299 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ca21095f-2156-4769-851c-c06475e23f85-service-ca\") pod \"console-686777b996-krg22\" (UID: \"ca21095f-2156-4769-851c-c06475e23f85\") " pod="openshift-console/console-686777b996-krg22" Jul 10 02:17:49.073446 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:17:49.073424 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ca21095f-2156-4769-851c-c06475e23f85-oauth-serving-cert\") pod \"console-686777b996-krg22\" (UID: \"ca21095f-2156-4769-851c-c06475e23f85\") " pod="openshift-console/console-686777b996-krg22" Jul 10 02:17:49.073498 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:17:49.073430 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca21095f-2156-4769-851c-c06475e23f85-trusted-ca-bundle\") pod \"console-686777b996-krg22\" (UID: \"ca21095f-2156-4769-851c-c06475e23f85\") " pod="openshift-console/console-686777b996-krg22" Jul 10 02:17:49.074935 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:17:49.074912 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca21095f-2156-4769-851c-c06475e23f85-console-serving-cert\") pod \"console-686777b996-krg22\" (UID: \"ca21095f-2156-4769-851c-c06475e23f85\") " pod="openshift-console/console-686777b996-krg22" Jul 10 02:17:49.075021 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:17:49.074987 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ca21095f-2156-4769-851c-c06475e23f85-console-oauth-config\") pod \"console-686777b996-krg22\" (UID: \"ca21095f-2156-4769-851c-c06475e23f85\") " pod="openshift-console/console-686777b996-krg22" Jul 10 02:17:49.081963 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:17:49.081933 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w5j4\" (UniqueName: \"kubernetes.io/projected/ca21095f-2156-4769-851c-c06475e23f85-kube-api-access-5w5j4\") pod \"console-686777b996-krg22\" (UID: \"ca21095f-2156-4769-851c-c06475e23f85\") " pod="openshift-console/console-686777b996-krg22" Jul 10 02:17:49.188061 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:17:49.188008 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-686777b996-krg22" Jul 10 02:17:49.319739 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:17:49.319713 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-686777b996-krg22"] Jul 10 02:17:49.321504 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:17:49.321479 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca21095f_2156_4769_851c_c06475e23f85.slice/crio-40d4b2638dd1cd77de3c69a8f460ba7ff412a0cc6cd07ca2e8fe5843581a4c71 WatchSource:0}: Error finding container 40d4b2638dd1cd77de3c69a8f460ba7ff412a0cc6cd07ca2e8fe5843581a4c71: Status 404 returned error can't find the container with id 40d4b2638dd1cd77de3c69a8f460ba7ff412a0cc6cd07ca2e8fe5843581a4c71 Jul 10 02:17:50.014369 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:17:50.014252 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:17:50.114795 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:17:50.114763 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:17:50.291408 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:17:50.291373 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-686777b996-krg22" event={"ID":"ca21095f-2156-4769-851c-c06475e23f85","Type":"ContainerStarted","Data":"78fad2300f78588f6891c91d60a64ec354af26e2834d6be1343e490588722155"} Jul 10 02:17:50.291562 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:17:50.291440 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-686777b996-krg22" event={"ID":"ca21095f-2156-4769-851c-c06475e23f85","Type":"ContainerStarted","Data":"40d4b2638dd1cd77de3c69a8f460ba7ff412a0cc6cd07ca2e8fe5843581a4c71"} Jul 10 02:17:50.311446 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:17:50.311401 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-686777b996-krg22" podStartSLOduration=2.311387999 podStartE2EDuration="2.311387999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:17:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:17:50.31075504 +0000 UTC m=+601.887032128" watchObservedRunningTime="2026-07-10 02:17:50.311387999 +0000 UTC m=+601.887665084" Jul 10 02:17:50.830812 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:17:50.830777 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:17:51.037500 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:17:51.037469 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:17:54.129854 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:17:54.129822 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:17:54.196788 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:17:54.196758 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:17:59.188964 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:17:59.188931 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-686777b996-krg22" Jul 10 02:17:59.189333 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:17:59.188987 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-686777b996-krg22" Jul 10 02:17:59.193684 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:17:59.193659 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-686777b996-krg22" Jul 10 02:17:59.337401 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:17:59.337375 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-686777b996-krg22" Jul 10 02:17:59.386750 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:17:59.386719 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75c57dfdcc-mbjmf"] Jul 10 02:17:59.548025 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:17:59.547996 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:17:59.646900 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:17:59.646868 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:17:59.747753 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:17:59.747729 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:18:05.618945 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:05.618911 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594bstx"] Jul 10 02:18:05.623426 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:05.623409 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594bstx" Jul 10 02:18:05.626404 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:05.626382 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 10 02:18:05.628336 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:05.628126 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-dc49t\"" Jul 10 02:18:05.628336 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:05.628160 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 10 02:18:05.630827 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:05.630803 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594bstx"] Jul 10 02:18:05.713355 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:05.713326 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/92d2a4bc-f048-4da4-a974-4787843d0874-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594bstx\" (UID: \"92d2a4bc-f048-4da4-a974-4787843d0874\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594bstx" Jul 10 02:18:05.713489 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:05.713366 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcsl8\" (UniqueName: \"kubernetes.io/projected/92d2a4bc-f048-4da4-a974-4787843d0874-kube-api-access-jcsl8\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594bstx\" (UID: \"92d2a4bc-f048-4da4-a974-4787843d0874\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594bstx" Jul 10 02:18:05.713489 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:05.713403 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/92d2a4bc-f048-4da4-a974-4787843d0874-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594bstx\" (UID: \"92d2a4bc-f048-4da4-a974-4787843d0874\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594bstx" Jul 10 02:18:05.814025 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:05.813993 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/92d2a4bc-f048-4da4-a974-4787843d0874-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594bstx\" (UID: \"92d2a4bc-f048-4da4-a974-4787843d0874\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594bstx" Jul 10 02:18:05.814186 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:05.814032 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcsl8\" (UniqueName: \"kubernetes.io/projected/92d2a4bc-f048-4da4-a974-4787843d0874-kube-api-access-jcsl8\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594bstx\" (UID: \"92d2a4bc-f048-4da4-a974-4787843d0874\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594bstx" Jul 10 02:18:05.814186 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:05.814163 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/92d2a4bc-f048-4da4-a974-4787843d0874-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594bstx\" (UID: \"92d2a4bc-f048-4da4-a974-4787843d0874\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594bstx" Jul 10 02:18:05.814399 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:05.814383 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/92d2a4bc-f048-4da4-a974-4787843d0874-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594bstx\" (UID: \"92d2a4bc-f048-4da4-a974-4787843d0874\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594bstx" Jul 10 02:18:05.814486 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:05.814472 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/92d2a4bc-f048-4da4-a974-4787843d0874-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594bstx\" (UID: \"92d2a4bc-f048-4da4-a974-4787843d0874\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594bstx" Jul 10 02:18:05.824007 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:05.823981 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcsl8\" (UniqueName: \"kubernetes.io/projected/92d2a4bc-f048-4da4-a974-4787843d0874-kube-api-access-jcsl8\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594bstx\" (UID: \"92d2a4bc-f048-4da4-a974-4787843d0874\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594bstx" Jul 10 02:18:05.934265 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:05.934204 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594bstx" Jul 10 02:18:06.063984 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:06.063958 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594bstx"] Jul 10 02:18:06.065533 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:18:06.065506 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92d2a4bc_f048_4da4_a974_4787843d0874.slice/crio-baab9c64871d864e88c848e52c4625b75fd6d0c8d06f5855ec820139d46516eb WatchSource:0}: Error finding container baab9c64871d864e88c848e52c4625b75fd6d0c8d06f5855ec820139d46516eb: Status 404 returned error can't find the container with id baab9c64871d864e88c848e52c4625b75fd6d0c8d06f5855ec820139d46516eb Jul 10 02:18:06.366731 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:06.366697 2566 generic.go:356] "Generic (PLEG): container finished" podID="92d2a4bc-f048-4da4-a974-4787843d0874" containerID="922dcca30e11458975e9ddbab6ed519a499f283a2e98a252b3357ec630d39669" exitCode=0 Jul 10 02:18:06.366840 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:06.366779 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594bstx" event={"ID":"92d2a4bc-f048-4da4-a974-4787843d0874","Type":"ContainerDied","Data":"922dcca30e11458975e9ddbab6ed519a499f283a2e98a252b3357ec630d39669"} Jul 10 02:18:06.366840 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:06.366820 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594bstx" event={"ID":"92d2a4bc-f048-4da4-a974-4787843d0874","Type":"ContainerStarted","Data":"baab9c64871d864e88c848e52c4625b75fd6d0c8d06f5855ec820139d46516eb"} Jul 10 02:18:06.418879 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:06.418853 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq9xc"] Jul 10 02:18:06.422240 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:06.422224 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq9xc" Jul 10 02:18:06.431546 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:06.431528 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq9xc"] Jul 10 02:18:06.521439 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:06.521416 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44f94038-e230-4433-985a-91a3383be807-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq9xc\" (UID: \"44f94038-e230-4433-985a-91a3383be807\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq9xc" Jul 10 02:18:06.521555 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:06.521465 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44f94038-e230-4433-985a-91a3383be807-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq9xc\" (UID: \"44f94038-e230-4433-985a-91a3383be807\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq9xc" Jul 10 02:18:06.521623 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:06.521552 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds95q\" (UniqueName: \"kubernetes.io/projected/44f94038-e230-4433-985a-91a3383be807-kube-api-access-ds95q\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq9xc\" (UID: \"44f94038-e230-4433-985a-91a3383be807\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq9xc" Jul 10 02:18:06.622815 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:06.622756 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds95q\" (UniqueName: \"kubernetes.io/projected/44f94038-e230-4433-985a-91a3383be807-kube-api-access-ds95q\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq9xc\" (UID: \"44f94038-e230-4433-985a-91a3383be807\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq9xc" Jul 10 02:18:06.622815 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:06.622797 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44f94038-e230-4433-985a-91a3383be807-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq9xc\" (UID: \"44f94038-e230-4433-985a-91a3383be807\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq9xc" Jul 10 02:18:06.623133 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:06.622917 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44f94038-e230-4433-985a-91a3383be807-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq9xc\" (UID: \"44f94038-e230-4433-985a-91a3383be807\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq9xc" Jul 10 02:18:06.623133 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:06.623085 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44f94038-e230-4433-985a-91a3383be807-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq9xc\" (UID: \"44f94038-e230-4433-985a-91a3383be807\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq9xc" Jul 10 02:18:06.623237 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:06.623219 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44f94038-e230-4433-985a-91a3383be807-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq9xc\" (UID: \"44f94038-e230-4433-985a-91a3383be807\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq9xc" Jul 10 02:18:06.637491 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:06.637470 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds95q\" (UniqueName: \"kubernetes.io/projected/44f94038-e230-4433-985a-91a3383be807-kube-api-access-ds95q\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq9xc\" (UID: \"44f94038-e230-4433-985a-91a3383be807\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq9xc" Jul 10 02:18:06.732284 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:06.732261 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq9xc" Jul 10 02:18:06.867681 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:06.867655 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq9xc"] Jul 10 02:18:06.869570 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:18:06.869539 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44f94038_e230_4433_985a_91a3383be807.slice/crio-6a184bac4be973655fa538944e0b3000a7d870757997761580442f88cfaf88d4 WatchSource:0}: Error finding container 6a184bac4be973655fa538944e0b3000a7d870757997761580442f88cfaf88d4: Status 404 returned error can't find the container with id 6a184bac4be973655fa538944e0b3000a7d870757997761580442f88cfaf88d4 Jul 10 02:18:07.221201 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:07.221177 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cfkcm"] Jul 10 02:18:07.224544 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:07.224528 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cfkcm" Jul 10 02:18:07.232335 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:07.232313 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cfkcm"] Jul 10 02:18:07.329703 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:07.329677 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f870d1d5-2091-49f5-b738-bcf5fa6bcfed-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cfkcm\" (UID: \"f870d1d5-2091-49f5-b738-bcf5fa6bcfed\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cfkcm" Jul 10 02:18:07.329810 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:07.329716 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f870d1d5-2091-49f5-b738-bcf5fa6bcfed-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cfkcm\" (UID: \"f870d1d5-2091-49f5-b738-bcf5fa6bcfed\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cfkcm" Jul 10 02:18:07.329810 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:07.329738 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mftws\" (UniqueName: \"kubernetes.io/projected/f870d1d5-2091-49f5-b738-bcf5fa6bcfed-kube-api-access-mftws\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cfkcm\" (UID: \"f870d1d5-2091-49f5-b738-bcf5fa6bcfed\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cfkcm" Jul 10 02:18:07.373698 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:07.373666 2566 generic.go:356] "Generic (PLEG): container finished" podID="92d2a4bc-f048-4da4-a974-4787843d0874" containerID="c0a1a1060b54f2a8e4a951e0f4b30b8333f9c6c63b4b385211cbbee041f8858a" exitCode=0 Jul 10 02:18:07.373789 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:07.373748 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594bstx" event={"ID":"92d2a4bc-f048-4da4-a974-4787843d0874","Type":"ContainerDied","Data":"c0a1a1060b54f2a8e4a951e0f4b30b8333f9c6c63b4b385211cbbee041f8858a"} Jul 10 02:18:07.375255 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:07.375235 2566 generic.go:356] "Generic (PLEG): container finished" podID="44f94038-e230-4433-985a-91a3383be807" containerID="aa989e8df5ae606b39944c4cb85e44423ec084083518adb2bce59ab8c8e9b877" exitCode=0 Jul 10 02:18:07.375320 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:07.375275 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq9xc" event={"ID":"44f94038-e230-4433-985a-91a3383be807","Type":"ContainerDied","Data":"aa989e8df5ae606b39944c4cb85e44423ec084083518adb2bce59ab8c8e9b877"} Jul 10 02:18:07.375364 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:07.375318 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq9xc" event={"ID":"44f94038-e230-4433-985a-91a3383be807","Type":"ContainerStarted","Data":"6a184bac4be973655fa538944e0b3000a7d870757997761580442f88cfaf88d4"} Jul 10 02:18:07.430984 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:07.430962 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f870d1d5-2091-49f5-b738-bcf5fa6bcfed-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cfkcm\" (UID: \"f870d1d5-2091-49f5-b738-bcf5fa6bcfed\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cfkcm" Jul 10 02:18:07.431074 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:07.430997 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f870d1d5-2091-49f5-b738-bcf5fa6bcfed-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cfkcm\" (UID: \"f870d1d5-2091-49f5-b738-bcf5fa6bcfed\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cfkcm" Jul 10 02:18:07.431074 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:07.431030 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mftws\" (UniqueName: \"kubernetes.io/projected/f870d1d5-2091-49f5-b738-bcf5fa6bcfed-kube-api-access-mftws\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cfkcm\" (UID: \"f870d1d5-2091-49f5-b738-bcf5fa6bcfed\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cfkcm" Jul 10 02:18:07.431329 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:07.431301 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f870d1d5-2091-49f5-b738-bcf5fa6bcfed-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cfkcm\" (UID: \"f870d1d5-2091-49f5-b738-bcf5fa6bcfed\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cfkcm" Jul 10 02:18:07.431425 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:07.431345 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f870d1d5-2091-49f5-b738-bcf5fa6bcfed-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cfkcm\" (UID: \"f870d1d5-2091-49f5-b738-bcf5fa6bcfed\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cfkcm" Jul 10 02:18:07.441251 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:07.441218 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mftws\" (UniqueName: \"kubernetes.io/projected/f870d1d5-2091-49f5-b738-bcf5fa6bcfed-kube-api-access-mftws\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cfkcm\" (UID: \"f870d1d5-2091-49f5-b738-bcf5fa6bcfed\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cfkcm" Jul 10 02:18:07.561244 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:07.561218 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cfkcm" Jul 10 02:18:07.692201 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:07.692179 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cfkcm"] Jul 10 02:18:07.694301 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:18:07.694264 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf870d1d5_2091_49f5_b738_bcf5fa6bcfed.slice/crio-ea6b7d4f4fb3535471d5e56dca2c09a1ab2a3b3ec3c5e40a0587ff269f7ad112 WatchSource:0}: Error finding container ea6b7d4f4fb3535471d5e56dca2c09a1ab2a3b3ec3c5e40a0587ff269f7ad112: Status 404 returned error can't find the container with id ea6b7d4f4fb3535471d5e56dca2c09a1ab2a3b3ec3c5e40a0587ff269f7ad112 Jul 10 02:18:07.821481 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:07.821453 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02rdkd"] Jul 10 02:18:07.824966 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:07.824951 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02rdkd" Jul 10 02:18:07.833980 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:07.833956 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02rdkd"] Jul 10 02:18:07.865512 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:07.865486 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:18:07.921844 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:07.921763 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:18:07.935477 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:07.935448 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0545a39d-110a-43eb-9e80-9d1ca3643afd-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02rdkd\" (UID: \"0545a39d-110a-43eb-9e80-9d1ca3643afd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02rdkd" Jul 10 02:18:07.935647 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:07.935529 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sx29\" (UniqueName: \"kubernetes.io/projected/0545a39d-110a-43eb-9e80-9d1ca3643afd-kube-api-access-2sx29\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02rdkd\" (UID: \"0545a39d-110a-43eb-9e80-9d1ca3643afd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02rdkd" Jul 10 02:18:07.935647 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:07.935567 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0545a39d-110a-43eb-9e80-9d1ca3643afd-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02rdkd\" (UID: \"0545a39d-110a-43eb-9e80-9d1ca3643afd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02rdkd" Jul 10 02:18:08.036196 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:08.036162 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sx29\" (UniqueName: \"kubernetes.io/projected/0545a39d-110a-43eb-9e80-9d1ca3643afd-kube-api-access-2sx29\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02rdkd\" (UID: \"0545a39d-110a-43eb-9e80-9d1ca3643afd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02rdkd" Jul 10 02:18:08.036315 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:08.036211 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0545a39d-110a-43eb-9e80-9d1ca3643afd-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02rdkd\" (UID: \"0545a39d-110a-43eb-9e80-9d1ca3643afd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02rdkd" Jul 10 02:18:08.036315 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:08.036265 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0545a39d-110a-43eb-9e80-9d1ca3643afd-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02rdkd\" (UID: \"0545a39d-110a-43eb-9e80-9d1ca3643afd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02rdkd" Jul 10 02:18:08.036587 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:08.036570 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0545a39d-110a-43eb-9e80-9d1ca3643afd-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02rdkd\" (UID: \"0545a39d-110a-43eb-9e80-9d1ca3643afd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02rdkd" Jul 10 02:18:08.036740 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:08.036718 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0545a39d-110a-43eb-9e80-9d1ca3643afd-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02rdkd\" (UID: \"0545a39d-110a-43eb-9e80-9d1ca3643afd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02rdkd" Jul 10 02:18:08.047010 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:08.046983 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sx29\" (UniqueName: \"kubernetes.io/projected/0545a39d-110a-43eb-9e80-9d1ca3643afd-kube-api-access-2sx29\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02rdkd\" (UID: \"0545a39d-110a-43eb-9e80-9d1ca3643afd\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02rdkd" Jul 10 02:18:08.163849 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:08.163817 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02rdkd" Jul 10 02:18:08.297218 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:08.297196 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02rdkd"] Jul 10 02:18:08.298932 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:18:08.298906 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0545a39d_110a_43eb_9e80_9d1ca3643afd.slice/crio-782c8cbfa57eabd382fede7159a99204b3176de3dfcf6ebdad9ddeaa5cfc4b0d WatchSource:0}: Error finding container 782c8cbfa57eabd382fede7159a99204b3176de3dfcf6ebdad9ddeaa5cfc4b0d: Status 404 returned error can't find the container with id 782c8cbfa57eabd382fede7159a99204b3176de3dfcf6ebdad9ddeaa5cfc4b0d Jul 10 02:18:08.382115 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:08.382077 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02rdkd" event={"ID":"0545a39d-110a-43eb-9e80-9d1ca3643afd","Type":"ContainerStarted","Data":"bf42bd3bddfdeba3caad08835f349cb66e9185ad3cc9a4405044333217bb4777"} Jul 10 02:18:08.382290 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:08.382145 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02rdkd" event={"ID":"0545a39d-110a-43eb-9e80-9d1ca3643afd","Type":"ContainerStarted","Data":"782c8cbfa57eabd382fede7159a99204b3176de3dfcf6ebdad9ddeaa5cfc4b0d"} Jul 10 02:18:08.384426 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:08.384397 2566 generic.go:356] "Generic (PLEG): container finished" podID="92d2a4bc-f048-4da4-a974-4787843d0874" containerID="e10d484a82fa7b95af86ce04e3520d9b66c14b1b21b2eb23806fb030e4291026" exitCode=0 Jul 10 02:18:08.384549 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:08.384482 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594bstx" event={"ID":"92d2a4bc-f048-4da4-a974-4787843d0874","Type":"ContainerDied","Data":"e10d484a82fa7b95af86ce04e3520d9b66c14b1b21b2eb23806fb030e4291026"} Jul 10 02:18:08.386269 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:08.386243 2566 generic.go:356] "Generic (PLEG): container finished" podID="44f94038-e230-4433-985a-91a3383be807" containerID="21c927a480f5c35f87435101035918dff9b44991fa1ed41249596d707b60857b" exitCode=0 Jul 10 02:18:08.386371 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:08.386270 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq9xc" event={"ID":"44f94038-e230-4433-985a-91a3383be807","Type":"ContainerDied","Data":"21c927a480f5c35f87435101035918dff9b44991fa1ed41249596d707b60857b"} Jul 10 02:18:08.388064 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:08.388045 2566 generic.go:356] "Generic (PLEG): container finished" podID="f870d1d5-2091-49f5-b738-bcf5fa6bcfed" containerID="711b38c18d8520b1b4e5fbc229d42d2a7cb780a89c0ccb18edb3d948a896aeee" exitCode=0 Jul 10 02:18:08.388133 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:08.388077 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cfkcm" event={"ID":"f870d1d5-2091-49f5-b738-bcf5fa6bcfed","Type":"ContainerDied","Data":"711b38c18d8520b1b4e5fbc229d42d2a7cb780a89c0ccb18edb3d948a896aeee"} Jul 10 02:18:08.388133 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:08.388102 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cfkcm" event={"ID":"f870d1d5-2091-49f5-b738-bcf5fa6bcfed","Type":"ContainerStarted","Data":"ea6b7d4f4fb3535471d5e56dca2c09a1ab2a3b3ec3c5e40a0587ff269f7ad112"} Jul 10 02:18:09.395184 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:09.395151 2566 generic.go:356] "Generic (PLEG): container finished" podID="44f94038-e230-4433-985a-91a3383be807" containerID="1beb79ed744858fa74099d28a6b881a3e2abefc32d1169a50e21eef435997901" exitCode=0 Jul 10 02:18:09.395531 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:09.395236 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq9xc" event={"ID":"44f94038-e230-4433-985a-91a3383be807","Type":"ContainerDied","Data":"1beb79ed744858fa74099d28a6b881a3e2abefc32d1169a50e21eef435997901"} Jul 10 02:18:09.396893 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:09.396871 2566 generic.go:356] "Generic (PLEG): container finished" podID="f870d1d5-2091-49f5-b738-bcf5fa6bcfed" containerID="9e999d1c7bc2c659b46bcbf2219854b0061402da88d146052a50fa27d9889dd2" exitCode=0 Jul 10 02:18:09.396993 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:09.396921 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cfkcm" event={"ID":"f870d1d5-2091-49f5-b738-bcf5fa6bcfed","Type":"ContainerDied","Data":"9e999d1c7bc2c659b46bcbf2219854b0061402da88d146052a50fa27d9889dd2"} Jul 10 02:18:09.398487 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:09.398470 2566 generic.go:356] "Generic (PLEG): container finished" podID="0545a39d-110a-43eb-9e80-9d1ca3643afd" containerID="bf42bd3bddfdeba3caad08835f349cb66e9185ad3cc9a4405044333217bb4777" exitCode=0 Jul 10 02:18:09.398556 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:09.398528 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02rdkd" event={"ID":"0545a39d-110a-43eb-9e80-9d1ca3643afd","Type":"ContainerDied","Data":"bf42bd3bddfdeba3caad08835f349cb66e9185ad3cc9a4405044333217bb4777"} Jul 10 02:18:09.531370 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:09.531337 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594bstx" Jul 10 02:18:09.650416 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:09.650392 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/92d2a4bc-f048-4da4-a974-4787843d0874-util\") pod \"92d2a4bc-f048-4da4-a974-4787843d0874\" (UID: \"92d2a4bc-f048-4da4-a974-4787843d0874\") " Jul 10 02:18:09.650578 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:09.650421 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/92d2a4bc-f048-4da4-a974-4787843d0874-bundle\") pod \"92d2a4bc-f048-4da4-a974-4787843d0874\" (UID: \"92d2a4bc-f048-4da4-a974-4787843d0874\") " Jul 10 02:18:09.650578 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:09.650478 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jcsl8\" (UniqueName: \"kubernetes.io/projected/92d2a4bc-f048-4da4-a974-4787843d0874-kube-api-access-jcsl8\") pod \"92d2a4bc-f048-4da4-a974-4787843d0874\" (UID: \"92d2a4bc-f048-4da4-a974-4787843d0874\") " Jul 10 02:18:09.650922 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:09.650897 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92d2a4bc-f048-4da4-a974-4787843d0874-bundle" (OuterVolumeSpecName: "bundle") pod "92d2a4bc-f048-4da4-a974-4787843d0874" (UID: "92d2a4bc-f048-4da4-a974-4787843d0874"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 02:18:09.652515 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:09.652493 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92d2a4bc-f048-4da4-a974-4787843d0874-kube-api-access-jcsl8" (OuterVolumeSpecName: "kube-api-access-jcsl8") pod "92d2a4bc-f048-4da4-a974-4787843d0874" (UID: "92d2a4bc-f048-4da4-a974-4787843d0874"). InnerVolumeSpecName "kube-api-access-jcsl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 02:18:09.655703 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:09.655666 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92d2a4bc-f048-4da4-a974-4787843d0874-util" (OuterVolumeSpecName: "util") pod "92d2a4bc-f048-4da4-a974-4787843d0874" (UID: "92d2a4bc-f048-4da4-a974-4787843d0874"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 02:18:09.751663 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:09.751589 2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/92d2a4bc-f048-4da4-a974-4787843d0874-util\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:18:09.751663 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:09.751624 2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/92d2a4bc-f048-4da4-a974-4787843d0874-bundle\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:18:09.751663 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:09.751632 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcsl8\" (UniqueName: \"kubernetes.io/projected/92d2a4bc-f048-4da4-a974-4787843d0874-kube-api-access-jcsl8\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:18:10.405225 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:10.405204 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594bstx" Jul 10 02:18:10.405559 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:10.405224 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594bstx" event={"ID":"92d2a4bc-f048-4da4-a974-4787843d0874","Type":"ContainerDied","Data":"baab9c64871d864e88c848e52c4625b75fd6d0c8d06f5855ec820139d46516eb"} Jul 10 02:18:10.405559 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:10.405273 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="baab9c64871d864e88c848e52c4625b75fd6d0c8d06f5855ec820139d46516eb" Jul 10 02:18:10.407394 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:10.407374 2566 generic.go:356] "Generic (PLEG): container finished" podID="f870d1d5-2091-49f5-b738-bcf5fa6bcfed" containerID="026078599c3158282999770c183492f84ca4db255b2e9b5ddd0627d9405f8a1b" exitCode=0 Jul 10 02:18:10.407499 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:10.407463 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cfkcm" event={"ID":"f870d1d5-2091-49f5-b738-bcf5fa6bcfed","Type":"ContainerDied","Data":"026078599c3158282999770c183492f84ca4db255b2e9b5ddd0627d9405f8a1b"} Jul 10 02:18:10.409111 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:10.409088 2566 generic.go:356] "Generic (PLEG): container finished" podID="0545a39d-110a-43eb-9e80-9d1ca3643afd" containerID="1e6d99444aacc36ef9b1329fa0329de538e9550eb1825a71aea9951324ac7497" exitCode=0 Jul 10 02:18:10.409192 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:10.409112 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02rdkd" event={"ID":"0545a39d-110a-43eb-9e80-9d1ca3643afd","Type":"ContainerDied","Data":"1e6d99444aacc36ef9b1329fa0329de538e9550eb1825a71aea9951324ac7497"} Jul 10 02:18:10.547736 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:10.547716 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq9xc" Jul 10 02:18:10.657310 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:10.657284 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44f94038-e230-4433-985a-91a3383be807-bundle\") pod \"44f94038-e230-4433-985a-91a3383be807\" (UID: \"44f94038-e230-4433-985a-91a3383be807\") " Jul 10 02:18:10.657450 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:10.657371 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44f94038-e230-4433-985a-91a3383be807-util\") pod \"44f94038-e230-4433-985a-91a3383be807\" (UID: \"44f94038-e230-4433-985a-91a3383be807\") " Jul 10 02:18:10.657450 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:10.657402 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ds95q\" (UniqueName: \"kubernetes.io/projected/44f94038-e230-4433-985a-91a3383be807-kube-api-access-ds95q\") pod \"44f94038-e230-4433-985a-91a3383be807\" (UID: \"44f94038-e230-4433-985a-91a3383be807\") " Jul 10 02:18:10.658028 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:10.658001 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f94038-e230-4433-985a-91a3383be807-bundle" (OuterVolumeSpecName: "bundle") pod "44f94038-e230-4433-985a-91a3383be807" (UID: "44f94038-e230-4433-985a-91a3383be807"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 02:18:10.659431 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:10.659412 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44f94038-e230-4433-985a-91a3383be807-kube-api-access-ds95q" (OuterVolumeSpecName: "kube-api-access-ds95q") pod "44f94038-e230-4433-985a-91a3383be807" (UID: "44f94038-e230-4433-985a-91a3383be807"). InnerVolumeSpecName "kube-api-access-ds95q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 02:18:10.662537 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:10.662517 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f94038-e230-4433-985a-91a3383be807-util" (OuterVolumeSpecName: "util") pod "44f94038-e230-4433-985a-91a3383be807" (UID: "44f94038-e230-4433-985a-91a3383be807"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 02:18:10.758292 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:10.758262 2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44f94038-e230-4433-985a-91a3383be807-util\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:18:10.758292 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:10.758288 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds95q\" (UniqueName: \"kubernetes.io/projected/44f94038-e230-4433-985a-91a3383be807-kube-api-access-ds95q\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:18:10.758292 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:10.758298 2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44f94038-e230-4433-985a-91a3383be807-bundle\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:18:11.416320 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:11.416284 2566 generic.go:356] "Generic (PLEG): container finished" podID="0545a39d-110a-43eb-9e80-9d1ca3643afd" containerID="2bcf70f97e33944837d462e7d00971e9f09c055c238f74f4823073574f214899" exitCode=0 Jul 10 02:18:11.416767 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:11.416368 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02rdkd" event={"ID":"0545a39d-110a-43eb-9e80-9d1ca3643afd","Type":"ContainerDied","Data":"2bcf70f97e33944837d462e7d00971e9f09c055c238f74f4823073574f214899"} Jul 10 02:18:11.418198 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:11.418179 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq9xc" Jul 10 02:18:11.418289 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:11.418193 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq9xc" event={"ID":"44f94038-e230-4433-985a-91a3383be807","Type":"ContainerDied","Data":"6a184bac4be973655fa538944e0b3000a7d870757997761580442f88cfaf88d4"} Jul 10 02:18:11.418289 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:11.418230 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a184bac4be973655fa538944e0b3000a7d870757997761580442f88cfaf88d4" Jul 10 02:18:11.547526 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:11.547498 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cfkcm" Jul 10 02:18:11.665126 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:11.665095 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f870d1d5-2091-49f5-b738-bcf5fa6bcfed-util\") pod \"f870d1d5-2091-49f5-b738-bcf5fa6bcfed\" (UID: \"f870d1d5-2091-49f5-b738-bcf5fa6bcfed\") " Jul 10 02:18:11.665286 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:11.665183 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mftws\" (UniqueName: \"kubernetes.io/projected/f870d1d5-2091-49f5-b738-bcf5fa6bcfed-kube-api-access-mftws\") pod \"f870d1d5-2091-49f5-b738-bcf5fa6bcfed\" (UID: \"f870d1d5-2091-49f5-b738-bcf5fa6bcfed\") " Jul 10 02:18:11.665286 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:11.665218 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f870d1d5-2091-49f5-b738-bcf5fa6bcfed-bundle\") pod \"f870d1d5-2091-49f5-b738-bcf5fa6bcfed\" (UID: \"f870d1d5-2091-49f5-b738-bcf5fa6bcfed\") " Jul 10 02:18:11.665820 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:11.665798 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f870d1d5-2091-49f5-b738-bcf5fa6bcfed-bundle" (OuterVolumeSpecName: "bundle") pod "f870d1d5-2091-49f5-b738-bcf5fa6bcfed" (UID: "f870d1d5-2091-49f5-b738-bcf5fa6bcfed"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 02:18:11.667272 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:11.667222 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f870d1d5-2091-49f5-b738-bcf5fa6bcfed-kube-api-access-mftws" (OuterVolumeSpecName: "kube-api-access-mftws") pod "f870d1d5-2091-49f5-b738-bcf5fa6bcfed" (UID: "f870d1d5-2091-49f5-b738-bcf5fa6bcfed"). InnerVolumeSpecName "kube-api-access-mftws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 02:18:11.670353 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:11.670320 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f870d1d5-2091-49f5-b738-bcf5fa6bcfed-util" (OuterVolumeSpecName: "util") pod "f870d1d5-2091-49f5-b738-bcf5fa6bcfed" (UID: "f870d1d5-2091-49f5-b738-bcf5fa6bcfed"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 02:18:11.766657 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:11.766632 2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f870d1d5-2091-49f5-b738-bcf5fa6bcfed-bundle\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:18:11.766657 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:11.766655 2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f870d1d5-2091-49f5-b738-bcf5fa6bcfed-util\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:18:11.766797 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:11.766664 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mftws\" (UniqueName: \"kubernetes.io/projected/f870d1d5-2091-49f5-b738-bcf5fa6bcfed-kube-api-access-mftws\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:18:12.424701 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:12.424665 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cfkcm" Jul 10 02:18:12.425117 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:12.424663 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cfkcm" event={"ID":"f870d1d5-2091-49f5-b738-bcf5fa6bcfed","Type":"ContainerDied","Data":"ea6b7d4f4fb3535471d5e56dca2c09a1ab2a3b3ec3c5e40a0587ff269f7ad112"} Jul 10 02:18:12.425117 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:12.424835 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea6b7d4f4fb3535471d5e56dca2c09a1ab2a3b3ec3c5e40a0587ff269f7ad112" Jul 10 02:18:12.555509 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:12.555486 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02rdkd" Jul 10 02:18:12.673883 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:12.673854 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0545a39d-110a-43eb-9e80-9d1ca3643afd-util\") pod \"0545a39d-110a-43eb-9e80-9d1ca3643afd\" (UID: \"0545a39d-110a-43eb-9e80-9d1ca3643afd\") " Jul 10 02:18:12.674021 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:12.673903 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2sx29\" (UniqueName: \"kubernetes.io/projected/0545a39d-110a-43eb-9e80-9d1ca3643afd-kube-api-access-2sx29\") pod \"0545a39d-110a-43eb-9e80-9d1ca3643afd\" (UID: \"0545a39d-110a-43eb-9e80-9d1ca3643afd\") " Jul 10 02:18:12.674021 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:12.673986 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0545a39d-110a-43eb-9e80-9d1ca3643afd-bundle\") pod \"0545a39d-110a-43eb-9e80-9d1ca3643afd\" (UID: \"0545a39d-110a-43eb-9e80-9d1ca3643afd\") " Jul 10 02:18:12.674489 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:12.674453 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0545a39d-110a-43eb-9e80-9d1ca3643afd-bundle" (OuterVolumeSpecName: "bundle") pod "0545a39d-110a-43eb-9e80-9d1ca3643afd" (UID: "0545a39d-110a-43eb-9e80-9d1ca3643afd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 02:18:12.675972 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:12.675924 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0545a39d-110a-43eb-9e80-9d1ca3643afd-kube-api-access-2sx29" (OuterVolumeSpecName: "kube-api-access-2sx29") pod "0545a39d-110a-43eb-9e80-9d1ca3643afd" (UID: "0545a39d-110a-43eb-9e80-9d1ca3643afd"). InnerVolumeSpecName "kube-api-access-2sx29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 02:18:12.682133 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:12.682109 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0545a39d-110a-43eb-9e80-9d1ca3643afd-util" (OuterVolumeSpecName: "util") pod "0545a39d-110a-43eb-9e80-9d1ca3643afd" (UID: "0545a39d-110a-43eb-9e80-9d1ca3643afd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 02:18:12.774952 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:12.774917 2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0545a39d-110a-43eb-9e80-9d1ca3643afd-bundle\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:18:12.774952 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:12.774948 2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0545a39d-110a-43eb-9e80-9d1ca3643afd-util\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:18:12.774952 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:12.774958 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sx29\" (UniqueName: \"kubernetes.io/projected/0545a39d-110a-43eb-9e80-9d1ca3643afd-kube-api-access-2sx29\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:18:13.431505 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:13.431475 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02rdkd" Jul 10 02:18:13.431901 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:13.431474 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02rdkd" event={"ID":"0545a39d-110a-43eb-9e80-9d1ca3643afd","Type":"ContainerDied","Data":"782c8cbfa57eabd382fede7159a99204b3176de3dfcf6ebdad9ddeaa5cfc4b0d"} Jul 10 02:18:13.431901 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:13.431630 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="782c8cbfa57eabd382fede7159a99204b3176de3dfcf6ebdad9ddeaa5cfc4b0d" Jul 10 02:18:19.864908 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:19.864865 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:18:23.470947 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:23.470907 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:18:23.591651 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:23.591619 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:18:24.413867 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:24.413805 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-75c57dfdcc-mbjmf" podUID="a91a4edc-b8be-4cf7-a86d-3177af27d334" containerName="console" containerID="cri-o://c256828ff37de6ea61f31b346c3c83dcaae0a52e86073df37a869299f8cc2e6f" gracePeriod=15 Jul 10 02:18:24.657456 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:24.657428 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75c57dfdcc-mbjmf" Jul 10 02:18:24.773914 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:24.773890 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a91a4edc-b8be-4cf7-a86d-3177af27d334-console-config\") pod \"a91a4edc-b8be-4cf7-a86d-3177af27d334\" (UID: \"a91a4edc-b8be-4cf7-a86d-3177af27d334\") " Jul 10 02:18:24.774076 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:24.773922 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a91a4edc-b8be-4cf7-a86d-3177af27d334-service-ca\") pod \"a91a4edc-b8be-4cf7-a86d-3177af27d334\" (UID: \"a91a4edc-b8be-4cf7-a86d-3177af27d334\") " Jul 10 02:18:24.774076 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:24.773971 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a91a4edc-b8be-4cf7-a86d-3177af27d334-trusted-ca-bundle\") pod \"a91a4edc-b8be-4cf7-a86d-3177af27d334\" (UID: \"a91a4edc-b8be-4cf7-a86d-3177af27d334\") " Jul 10 02:18:24.774076 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:24.774035 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a91a4edc-b8be-4cf7-a86d-3177af27d334-console-serving-cert\") pod \"a91a4edc-b8be-4cf7-a86d-3177af27d334\" (UID: \"a91a4edc-b8be-4cf7-a86d-3177af27d334\") " Jul 10 02:18:24.774251 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:24.774193 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgx6p\" (UniqueName: \"kubernetes.io/projected/a91a4edc-b8be-4cf7-a86d-3177af27d334-kube-api-access-jgx6p\") pod \"a91a4edc-b8be-4cf7-a86d-3177af27d334\" (UID: \"a91a4edc-b8be-4cf7-a86d-3177af27d334\") " Jul 10 02:18:24.774251 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:24.774241 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a91a4edc-b8be-4cf7-a86d-3177af27d334-oauth-serving-cert\") pod \"a91a4edc-b8be-4cf7-a86d-3177af27d334\" (UID: \"a91a4edc-b8be-4cf7-a86d-3177af27d334\") " Jul 10 02:18:24.774353 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:24.774286 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a91a4edc-b8be-4cf7-a86d-3177af27d334-console-oauth-config\") pod \"a91a4edc-b8be-4cf7-a86d-3177af27d334\" (UID: \"a91a4edc-b8be-4cf7-a86d-3177af27d334\") " Jul 10 02:18:24.774408 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:24.774359 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a91a4edc-b8be-4cf7-a86d-3177af27d334-console-config" (OuterVolumeSpecName: "console-config") pod "a91a4edc-b8be-4cf7-a86d-3177af27d334" (UID: "a91a4edc-b8be-4cf7-a86d-3177af27d334"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 02:18:24.774462 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:24.774441 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a91a4edc-b8be-4cf7-a86d-3177af27d334-service-ca" (OuterVolumeSpecName: "service-ca") pod "a91a4edc-b8be-4cf7-a86d-3177af27d334" (UID: "a91a4edc-b8be-4cf7-a86d-3177af27d334"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 02:18:24.774462 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:24.774448 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a91a4edc-b8be-4cf7-a86d-3177af27d334-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a91a4edc-b8be-4cf7-a86d-3177af27d334" (UID: "a91a4edc-b8be-4cf7-a86d-3177af27d334"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 02:18:24.774688 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:24.774659 2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a91a4edc-b8be-4cf7-a86d-3177af27d334-console-config\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:18:24.774688 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:24.774684 2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a91a4edc-b8be-4cf7-a86d-3177af27d334-service-ca\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:18:24.774820 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:24.774696 2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a91a4edc-b8be-4cf7-a86d-3177af27d334-trusted-ca-bundle\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:18:24.774877 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:24.774825 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a91a4edc-b8be-4cf7-a86d-3177af27d334-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a91a4edc-b8be-4cf7-a86d-3177af27d334" (UID: "a91a4edc-b8be-4cf7-a86d-3177af27d334"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 02:18:24.776277 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:24.776246 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a91a4edc-b8be-4cf7-a86d-3177af27d334-kube-api-access-jgx6p" (OuterVolumeSpecName: "kube-api-access-jgx6p") pod "a91a4edc-b8be-4cf7-a86d-3177af27d334" (UID: "a91a4edc-b8be-4cf7-a86d-3177af27d334"). InnerVolumeSpecName "kube-api-access-jgx6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 02:18:24.776396 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:24.776307 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a91a4edc-b8be-4cf7-a86d-3177af27d334-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a91a4edc-b8be-4cf7-a86d-3177af27d334" (UID: "a91a4edc-b8be-4cf7-a86d-3177af27d334"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 02:18:24.776476 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:24.776455 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a91a4edc-b8be-4cf7-a86d-3177af27d334-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a91a4edc-b8be-4cf7-a86d-3177af27d334" (UID: "a91a4edc-b8be-4cf7-a86d-3177af27d334"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 02:18:24.875994 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:24.875972 2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a91a4edc-b8be-4cf7-a86d-3177af27d334-console-serving-cert\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:18:24.875994 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:24.875992 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgx6p\" (UniqueName: \"kubernetes.io/projected/a91a4edc-b8be-4cf7-a86d-3177af27d334-kube-api-access-jgx6p\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:18:24.876144 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:24.876001 2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a91a4edc-b8be-4cf7-a86d-3177af27d334-oauth-serving-cert\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:18:24.876144 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:24.876009 2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a91a4edc-b8be-4cf7-a86d-3177af27d334-console-oauth-config\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:18:25.494320 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:25.494139 2566 generic.go:356] "Generic (PLEG): container finished" podID="a91a4edc-b8be-4cf7-a86d-3177af27d334" containerID="c256828ff37de6ea61f31b346c3c83dcaae0a52e86073df37a869299f8cc2e6f" exitCode=0 Jul 10 02:18:25.494320 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:25.494190 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75c57dfdcc-mbjmf" event={"ID":"a91a4edc-b8be-4cf7-a86d-3177af27d334","Type":"ContainerDied","Data":"c256828ff37de6ea61f31b346c3c83dcaae0a52e86073df37a869299f8cc2e6f"} Jul 10 02:18:25.494320 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:25.494208 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75c57dfdcc-mbjmf" Jul 10 02:18:25.494320 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:25.494261 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75c57dfdcc-mbjmf" event={"ID":"a91a4edc-b8be-4cf7-a86d-3177af27d334","Type":"ContainerDied","Data":"323fa33fa95191aa832ffe52a92f295d386edf267b63d484b51c09d83a9b2703"} Jul 10 02:18:25.494320 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:25.494316 2566 scope.go:117] "RemoveContainer" containerID="c256828ff37de6ea61f31b346c3c83dcaae0a52e86073df37a869299f8cc2e6f" Jul 10 02:18:25.503987 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:25.503965 2566 scope.go:117] "RemoveContainer" containerID="c256828ff37de6ea61f31b346c3c83dcaae0a52e86073df37a869299f8cc2e6f" Jul 10 02:18:25.504309 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:18:25.504279 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c256828ff37de6ea61f31b346c3c83dcaae0a52e86073df37a869299f8cc2e6f\": container with ID starting with c256828ff37de6ea61f31b346c3c83dcaae0a52e86073df37a869299f8cc2e6f not found: ID does not exist" containerID="c256828ff37de6ea61f31b346c3c83dcaae0a52e86073df37a869299f8cc2e6f" Jul 10 02:18:25.504458 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:25.504325 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c256828ff37de6ea61f31b346c3c83dcaae0a52e86073df37a869299f8cc2e6f"} err="failed to get container status \"c256828ff37de6ea61f31b346c3c83dcaae0a52e86073df37a869299f8cc2e6f\": rpc error: code = NotFound desc = could not find container \"c256828ff37de6ea61f31b346c3c83dcaae0a52e86073df37a869299f8cc2e6f\": container with ID starting with c256828ff37de6ea61f31b346c3c83dcaae0a52e86073df37a869299f8cc2e6f not found: ID does not exist" Jul 10 02:18:25.521686 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:25.521648 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75c57dfdcc-mbjmf"] Jul 10 02:18:25.525241 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:25.525211 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-75c57dfdcc-mbjmf"] Jul 10 02:18:27.010367 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:27.010338 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a91a4edc-b8be-4cf7-a86d-3177af27d334" path="/var/lib/kubelet/pods/a91a4edc-b8be-4cf7-a86d-3177af27d334/volumes" Jul 10 02:18:28.072846 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:28.072803 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:18:28.504885 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:28.504798 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:18:29.918696 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:29.918656 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:18:31.480799 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:31.480729 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:18:32.310019 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:32.309983 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:18:33.663136 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:33.663103 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-mnsp8"] Jul 10 02:18:33.663580 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:33.663480 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f870d1d5-2091-49f5-b738-bcf5fa6bcfed" containerName="extract" Jul 10 02:18:33.663580 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:33.663495 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f870d1d5-2091-49f5-b738-bcf5fa6bcfed" containerName="extract" Jul 10 02:18:33.663580 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:33.663505 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f870d1d5-2091-49f5-b738-bcf5fa6bcfed" containerName="pull" Jul 10 02:18:33.663580 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:33.663510 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f870d1d5-2091-49f5-b738-bcf5fa6bcfed" containerName="pull" Jul 10 02:18:33.663580 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:33.663524 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a91a4edc-b8be-4cf7-a86d-3177af27d334" containerName="console" Jul 10 02:18:33.663580 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:33.663529 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a91a4edc-b8be-4cf7-a86d-3177af27d334" containerName="console" Jul 10 02:18:33.663580 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:33.663535 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92d2a4bc-f048-4da4-a974-4787843d0874" containerName="util" Jul 10 02:18:33.663580 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:33.663540 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="92d2a4bc-f048-4da4-a974-4787843d0874" containerName="util" Jul 10 02:18:33.663580 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:33.663548 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44f94038-e230-4433-985a-91a3383be807" containerName="extract" Jul 10 02:18:33.663580 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:33.663553 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="44f94038-e230-4433-985a-91a3383be807" containerName="extract" Jul 10 02:18:33.663580 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:33.663560 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0545a39d-110a-43eb-9e80-9d1ca3643afd" containerName="extract" Jul 10 02:18:33.663580 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:33.663565 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0545a39d-110a-43eb-9e80-9d1ca3643afd" containerName="extract" Jul 10 02:18:33.663580 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:33.663571 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0545a39d-110a-43eb-9e80-9d1ca3643afd" containerName="util" Jul 10 02:18:33.663580 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:33.663576 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0545a39d-110a-43eb-9e80-9d1ca3643afd" containerName="util" Jul 10 02:18:33.663580 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:33.663582 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92d2a4bc-f048-4da4-a974-4787843d0874" containerName="extract" Jul 10 02:18:33.663580 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:33.663587 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="92d2a4bc-f048-4da4-a974-4787843d0874" containerName="extract" Jul 10 02:18:33.663580 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:33.663615 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44f94038-e230-4433-985a-91a3383be807" containerName="pull" Jul 10 02:18:33.664156 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:33.663621 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="44f94038-e230-4433-985a-91a3383be807" containerName="pull" Jul 10 02:18:33.664156 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:33.663627 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92d2a4bc-f048-4da4-a974-4787843d0874" containerName="pull" Jul 10 02:18:33.664156 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:33.663632 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="92d2a4bc-f048-4da4-a974-4787843d0874" containerName="pull" Jul 10 02:18:33.664156 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:33.663638 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0545a39d-110a-43eb-9e80-9d1ca3643afd" containerName="pull" Jul 10 02:18:33.664156 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:33.663642 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0545a39d-110a-43eb-9e80-9d1ca3643afd" containerName="pull" Jul 10 02:18:33.664156 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:33.663650 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44f94038-e230-4433-985a-91a3383be807" containerName="util" Jul 10 02:18:33.664156 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:33.663654 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="44f94038-e230-4433-985a-91a3383be807" containerName="util" Jul 10 02:18:33.664156 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:33.663660 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f870d1d5-2091-49f5-b738-bcf5fa6bcfed" containerName="util" Jul 10 02:18:33.664156 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:33.663664 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f870d1d5-2091-49f5-b738-bcf5fa6bcfed" containerName="util" Jul 10 02:18:33.664156 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:33.663711 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="92d2a4bc-f048-4da4-a974-4787843d0874" containerName="extract" Jul 10 02:18:33.664156 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:33.663718 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a91a4edc-b8be-4cf7-a86d-3177af27d334" containerName="console" Jul 10 02:18:33.664156 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:33.663724 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="44f94038-e230-4433-985a-91a3383be807" containerName="extract" Jul 10 02:18:33.664156 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:33.663731 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f870d1d5-2091-49f5-b738-bcf5fa6bcfed" containerName="extract" Jul 10 02:18:33.664156 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:33.663736 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0545a39d-110a-43eb-9e80-9d1ca3643afd" containerName="extract" Jul 10 02:18:33.667958 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:33.667940 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-mnsp8" Jul 10 02:18:33.671285 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:33.671261 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 10 02:18:33.671398 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:33.671305 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 10 02:18:33.671398 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:33.671342 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 10 02:18:33.671513 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:33.671449 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-zcmpw\"" Jul 10 02:18:33.677119 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:33.677098 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-mnsp8"] Jul 10 02:18:33.848365 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:33.848335 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hrlq\" (UniqueName: \"kubernetes.io/projected/61031f36-a534-40fc-b736-35cb1712ffd3-kube-api-access-5hrlq\") pod \"dns-operator-controller-manager-864dd78c4c-mnsp8\" (UID: \"61031f36-a534-40fc-b736-35cb1712ffd3\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-mnsp8" Jul 10 02:18:33.949625 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:33.949543 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hrlq\" (UniqueName: \"kubernetes.io/projected/61031f36-a534-40fc-b736-35cb1712ffd3-kube-api-access-5hrlq\") pod \"dns-operator-controller-manager-864dd78c4c-mnsp8\" (UID: \"61031f36-a534-40fc-b736-35cb1712ffd3\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-mnsp8" Jul 10 02:18:33.964568 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:33.964548 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hrlq\" (UniqueName: \"kubernetes.io/projected/61031f36-a534-40fc-b736-35cb1712ffd3-kube-api-access-5hrlq\") pod \"dns-operator-controller-manager-864dd78c4c-mnsp8\" (UID: \"61031f36-a534-40fc-b736-35cb1712ffd3\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-mnsp8" Jul 10 02:18:33.978366 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:33.978344 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-mnsp8" Jul 10 02:18:34.108067 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:34.108045 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-mnsp8"] Jul 10 02:18:34.109929 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:18:34.109894 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61031f36_a534_40fc_b736_35cb1712ffd3.slice/crio-ad8866830f41201106573e315aa2482124cb9d8968a9d3ffd454c1cb7abee1a8 WatchSource:0}: Error finding container ad8866830f41201106573e315aa2482124cb9d8968a9d3ffd454c1cb7abee1a8: Status 404 returned error can't find the container with id ad8866830f41201106573e315aa2482124cb9d8968a9d3ffd454c1cb7abee1a8 Jul 10 02:18:34.501686 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:34.501651 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:18:34.545228 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:34.545193 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-mnsp8" event={"ID":"61031f36-a534-40fc-b736-35cb1712ffd3","Type":"ContainerStarted","Data":"ad8866830f41201106573e315aa2482124cb9d8968a9d3ffd454c1cb7abee1a8"} Jul 10 02:18:36.471325 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:36.471281 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:18:36.558191 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:36.558148 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-mnsp8" event={"ID":"61031f36-a534-40fc-b736-35cb1712ffd3","Type":"ContainerStarted","Data":"1b95a0171dae16c1893647a7ba65f79848f60d536dc2bc7f62f113f1d9011044"} Jul 10 02:18:36.558358 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:36.558246 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-mnsp8" Jul 10 02:18:36.565286 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:36.565257 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:18:36.578995 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:36.578948 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-mnsp8" podStartSLOduration=1.430593011 podStartE2EDuration="3.578933046s" totalImagesPullingTime="2.148340035s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:18:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:18:36.576687918 +0000 UTC m=+648.152965024" watchObservedRunningTime="2026-07-10 02:18:36.578933046 +0000 UTC m=+648.155210131" Jul 10 02:18:37.288445 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:37.288410 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:18:37.542352 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:37.542269 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:18:39.867023 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:39.866990 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:18:39.934688 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:39.934652 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:18:41.093040 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:41.092996 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:18:41.193862 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:41.193829 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:18:47.552625 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:47.552564 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9wzjx"] Jul 10 02:18:47.556074 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:47.556051 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9wzjx" Jul 10 02:18:47.560763 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:47.560738 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-gg6f2\"" Jul 10 02:18:47.564143 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:47.564122 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-mnsp8" Jul 10 02:18:47.569932 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:47.569910 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9wzjx"] Jul 10 02:18:47.659810 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:47.659778 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gf78\" (UniqueName: \"kubernetes.io/projected/57b0005a-50f5-4a9e-ad28-e3651574a834-kube-api-access-5gf78\") pod \"limitador-operator-controller-manager-859d7b78bd-9wzjx\" (UID: \"57b0005a-50f5-4a9e-ad28-e3651574a834\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9wzjx" Jul 10 02:18:47.761185 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:47.761154 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gf78\" (UniqueName: \"kubernetes.io/projected/57b0005a-50f5-4a9e-ad28-e3651574a834-kube-api-access-5gf78\") pod \"limitador-operator-controller-manager-859d7b78bd-9wzjx\" (UID: \"57b0005a-50f5-4a9e-ad28-e3651574a834\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9wzjx" Jul 10 02:18:47.774813 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:47.774789 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gf78\" (UniqueName: \"kubernetes.io/projected/57b0005a-50f5-4a9e-ad28-e3651574a834-kube-api-access-5gf78\") pod \"limitador-operator-controller-manager-859d7b78bd-9wzjx\" (UID: \"57b0005a-50f5-4a9e-ad28-e3651574a834\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9wzjx" Jul 10 02:18:47.867898 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:47.867835 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9wzjx" Jul 10 02:18:48.003682 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:48.003660 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9wzjx"] Jul 10 02:18:48.005484 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:18:48.005455 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57b0005a_50f5_4a9e_ad28_e3651574a834.slice/crio-5004ebb7531c6bd71a13e2eb31f803be673a09c00c2bc2ba8a2cd8ee9a4a06d7 WatchSource:0}: Error finding container 5004ebb7531c6bd71a13e2eb31f803be673a09c00c2bc2ba8a2cd8ee9a4a06d7: Status 404 returned error can't find the container with id 5004ebb7531c6bd71a13e2eb31f803be673a09c00c2bc2ba8a2cd8ee9a4a06d7 Jul 10 02:18:48.617790 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:48.617742 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9wzjx" event={"ID":"57b0005a-50f5-4a9e-ad28-e3651574a834","Type":"ContainerStarted","Data":"5004ebb7531c6bd71a13e2eb31f803be673a09c00c2bc2ba8a2cd8ee9a4a06d7"} Jul 10 02:18:49.450033 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:18:49.449993 2566 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to pull image or OCI artifact: pull image err: copying system image from manifest list: reading blob sha256:2780920e5dbfbe103d03a583ed75345306e572ec5a48cb10361f046767d9f29a: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image" image="quay.io/kuadrant/limitador-operator:v0.17.1" Jul 10 02:18:49.450193 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:18:49.450135 2566 kuberuntime_manager.go:1459] "Unhandled Error" err="container manager start failed in pod limitador-operator-controller-manager-859d7b78bd-9wzjx_kuadrant-system(57b0005a-50f5-4a9e-ad28-e3651574a834): ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: reading blob sha256:2780920e5dbfbe103d03a583ed75345306e572ec5a48cb10361f046767d9f29a: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image" logger="UnhandledError" Jul 10 02:18:49.450193 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:18:49.450175 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with ErrImagePull: \"unable to pull image or OCI artifact: pull image err: copying system image from manifest list: reading blob sha256:2780920e5dbfbe103d03a583ed75345306e572ec5a48cb10361f046767d9f29a: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image\"" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9wzjx" podUID="57b0005a-50f5-4a9e-ad28-e3651574a834" Jul 10 02:18:49.625093 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:18:49.625066 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/kuadrant/limitador-operator:v0.17.1\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: reading blob sha256:2780920e5dbfbe103d03a583ed75345306e572ec5a48cb10361f046767d9f29a: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image\"" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9wzjx" podUID="57b0005a-50f5-4a9e-ad28-e3651574a834" Jul 10 02:18:49.882632 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:49.882533 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:18:49.983489 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:49.983453 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:18:50.480564 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:50.480535 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:18:50.583394 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:50.583359 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:18:54.313045 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:54.313004 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:18:54.410098 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:54.410065 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:18:59.475366 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:59.475335 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:18:59.580223 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:18:59.580194 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:19:02.692226 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:19:02.692148 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9wzjx" event={"ID":"57b0005a-50f5-4a9e-ad28-e3651574a834","Type":"ContainerStarted","Data":"ca3aff8466527ec6e0ffe6b4e6ae0359c9548ca9b3026ff6a87b50d5b64e602f"} Jul 10 02:19:02.692680 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:19:02.692359 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9wzjx" Jul 10 02:19:02.712697 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:19:02.712655 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9wzjx" podStartSLOduration=1.299333374 podStartE2EDuration="15.712641818s" totalImagesPullingTime="14.413308444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:18:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-07-10 02:19:02.71150826 +0000 UTC m=+674.287785346" watchObservedRunningTime="2026-07-10 02:19:02.712641818 +0000 UTC m=+674.288918904" Jul 10 02:19:02.866395 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:19:02.866363 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:19:02.968282 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:19:02.968210 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:19:11.476057 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:19:11.476016 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:19:11.575885 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:19:11.575852 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:19:13.699551 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:19:13.699517 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9wzjx" Jul 10 02:19:13.943197 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:19:13.943165 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:19:14.045478 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:19:14.045445 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:19:17.184151 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:19:17.184076 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:19:17.377754 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:19:17.377717 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:19:18.284850 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:19:18.284811 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:19:18.385624 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:19:18.385562 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:19:18.910937 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:19:18.910902 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:19:19.026532 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:19:19.026486 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:19:19.125213 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:19:19.125166 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:19:19.839558 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:19:19.839525 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:19:19.901293 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:19:19.901252 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:19:22.082445 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:19:22.082412 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:19:22.191344 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:19:22.191305 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:19:30.214758 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:19:30.214725 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:19:31.114210 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:19:31.114174 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:19:31.198246 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:19:31.198209 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:19:33.559330 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:19:33.559260 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:19:33.621585 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:19:33.621538 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:19:40.206927 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:19:40.206818 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:19:40.307633 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:19:40.307570 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:19:40.739808 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:19:40.739761 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:19:53.333318 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:19:53.333281 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:19:53.633977 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:19:53.633891 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:19:54.625176 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:19:54.625141 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:20:00.492267 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:20:00.492228 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:20:00.566224 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:20:00.566185 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:20:00.628655 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:20:00.628619 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:20:00.736190 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:20:00.736160 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:20:04.377540 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:20:04.377505 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:20:04.475557 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:20:04.475520 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:20:15.120476 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:20:15.120443 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:20:15.176182 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:20:15.176154 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:20:38.962901 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:20:38.962864 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:20:39.044890 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:20:39.044849 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:20:39.513560 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:20:39.513515 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:20:39.607339 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:20:39.607302 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:20:40.027548 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:20:40.027518 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:20:40.126936 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:20:40.126903 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:20:50.278124 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:20:50.278087 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:20:50.368570 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:20:50.368534 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:20:56.349498 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:20:56.349463 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l6zwm"] Jul 10 02:20:56.353189 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:20:56.353172 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l6zwm" Jul 10 02:20:56.355899 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:20:56.355879 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-qlqbf\"" Jul 10 02:20:56.357403 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:20:56.357376 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 10 02:20:56.357542 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:20:56.357378 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 10 02:20:56.357542 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:20:56.357423 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 10 02:20:56.364297 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:20:56.364279 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l6zwm"] Jul 10 02:20:56.388217 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:20:56.388188 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:20:56.418873 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:20:56.418850 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe1e3e8b-8a7d-4580-a413-5bbe6df30fe9-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l6zwm\" (UID: \"fe1e3e8b-8a7d-4580-a413-5bbe6df30fe9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l6zwm" Jul 10 02:20:56.418968 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:20:56.418890 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe1e3e8b-8a7d-4580-a413-5bbe6df30fe9-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l6zwm\" (UID: \"fe1e3e8b-8a7d-4580-a413-5bbe6df30fe9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l6zwm" Jul 10 02:20:56.418968 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:20:56.418909 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75ghb\" (UniqueName: \"kubernetes.io/projected/fe1e3e8b-8a7d-4580-a413-5bbe6df30fe9-kube-api-access-75ghb\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l6zwm\" (UID: \"fe1e3e8b-8a7d-4580-a413-5bbe6df30fe9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l6zwm" Jul 10 02:20:56.418968 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:20:56.418959 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe1e3e8b-8a7d-4580-a413-5bbe6df30fe9-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l6zwm\" (UID: \"fe1e3e8b-8a7d-4580-a413-5bbe6df30fe9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l6zwm" Jul 10 02:20:56.419091 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:20:56.419007 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe1e3e8b-8a7d-4580-a413-5bbe6df30fe9-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l6zwm\" (UID: \"fe1e3e8b-8a7d-4580-a413-5bbe6df30fe9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l6zwm" Jul 10 02:20:56.419091 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:20:56.419035 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe1e3e8b-8a7d-4580-a413-5bbe6df30fe9-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l6zwm\" (UID: \"fe1e3e8b-8a7d-4580-a413-5bbe6df30fe9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l6zwm" Jul 10 02:20:56.519555 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:20:56.519526 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe1e3e8b-8a7d-4580-a413-5bbe6df30fe9-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l6zwm\" (UID: \"fe1e3e8b-8a7d-4580-a413-5bbe6df30fe9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l6zwm" Jul 10 02:20:56.519707 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:20:56.519566 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe1e3e8b-8a7d-4580-a413-5bbe6df30fe9-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l6zwm\" (UID: \"fe1e3e8b-8a7d-4580-a413-5bbe6df30fe9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l6zwm" Jul 10 02:20:56.519707 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:20:56.519619 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe1e3e8b-8a7d-4580-a413-5bbe6df30fe9-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l6zwm\" (UID: \"fe1e3e8b-8a7d-4580-a413-5bbe6df30fe9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l6zwm" Jul 10 02:20:56.519707 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:20:56.519637 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75ghb\" (UniqueName: \"kubernetes.io/projected/fe1e3e8b-8a7d-4580-a413-5bbe6df30fe9-kube-api-access-75ghb\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l6zwm\" (UID: \"fe1e3e8b-8a7d-4580-a413-5bbe6df30fe9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l6zwm" Jul 10 02:20:56.519707 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:20:56.519662 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe1e3e8b-8a7d-4580-a413-5bbe6df30fe9-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l6zwm\" (UID: \"fe1e3e8b-8a7d-4580-a413-5bbe6df30fe9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l6zwm" Jul 10 02:20:56.519707 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:20:56.519688 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe1e3e8b-8a7d-4580-a413-5bbe6df30fe9-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l6zwm\" (UID: \"fe1e3e8b-8a7d-4580-a413-5bbe6df30fe9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l6zwm" Jul 10 02:20:56.520004 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:20:56.519980 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe1e3e8b-8a7d-4580-a413-5bbe6df30fe9-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l6zwm\" (UID: \"fe1e3e8b-8a7d-4580-a413-5bbe6df30fe9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l6zwm" Jul 10 02:20:56.520104 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:20:56.520004 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe1e3e8b-8a7d-4580-a413-5bbe6df30fe9-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l6zwm\" (UID: \"fe1e3e8b-8a7d-4580-a413-5bbe6df30fe9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l6zwm" Jul 10 02:20:56.520104 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:20:56.520040 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe1e3e8b-8a7d-4580-a413-5bbe6df30fe9-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l6zwm\" (UID: \"fe1e3e8b-8a7d-4580-a413-5bbe6df30fe9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l6zwm" Jul 10 02:20:56.521996 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:20:56.521974 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe1e3e8b-8a7d-4580-a413-5bbe6df30fe9-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l6zwm\" (UID: \"fe1e3e8b-8a7d-4580-a413-5bbe6df30fe9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l6zwm" Jul 10 02:20:56.522075 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:20:56.522059 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe1e3e8b-8a7d-4580-a413-5bbe6df30fe9-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l6zwm\" (UID: \"fe1e3e8b-8a7d-4580-a413-5bbe6df30fe9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l6zwm" Jul 10 02:20:56.528470 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:20:56.528446 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75ghb\" (UniqueName: \"kubernetes.io/projected/fe1e3e8b-8a7d-4580-a413-5bbe6df30fe9-kube-api-access-75ghb\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l6zwm\" (UID: \"fe1e3e8b-8a7d-4580-a413-5bbe6df30fe9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l6zwm" Jul 10 02:20:56.665006 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:20:56.664927 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l6zwm" Jul 10 02:20:56.801396 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:20:56.801370 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l6zwm"] Jul 10 02:20:56.805533 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:20:56.805502 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe1e3e8b_8a7d_4580_a413_5bbe6df30fe9.slice/crio-e6d8470ca2d54cb4ea10e4c4e7ae405b533527de10af999ea95d965d3d849816 WatchSource:0}: Error finding container e6d8470ca2d54cb4ea10e4c4e7ae405b533527de10af999ea95d965d3d849816: Status 404 returned error can't find the container with id e6d8470ca2d54cb4ea10e4c4e7ae405b533527de10af999ea95d965d3d849816 Jul 10 02:20:56.806830 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:20:56.806807 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 02:20:57.269282 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:20:57.269247 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l6zwm" event={"ID":"fe1e3e8b-8a7d-4580-a413-5bbe6df30fe9","Type":"ContainerStarted","Data":"e6d8470ca2d54cb4ea10e4c4e7ae405b533527de10af999ea95d965d3d849816"} Jul 10 02:20:58.276439 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:20:58.276408 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l6zwm" event={"ID":"fe1e3e8b-8a7d-4580-a413-5bbe6df30fe9","Type":"ContainerStarted","Data":"6935e14ef3d3e5f2e7d91a286b3be11a91d7aa407dff30ad46d859ca75649ed5"} Jul 10 02:20:58.295876 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:20:58.295832 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l6zwm" podStartSLOduration=1.144628593 podStartE2EDuration="2.295816233s" totalImagesPullingTime="1.15118764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:20:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:20:58.294401708 +0000 UTC m=+789.870678794" watchObservedRunningTime="2026-07-10 02:20:58.295816233 +0000 UTC m=+789.872093318" Jul 10 02:21:02.391725 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:21:02.391689 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:21:06.665968 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:21:06.665932 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l6zwm" Jul 10 02:21:06.666371 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:21:06.666081 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l6zwm" Jul 10 02:21:06.678528 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:21:06.678504 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l6zwm" Jul 10 02:21:07.332639 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:21:07.332607 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l6zwm" Jul 10 02:21:12.081172 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:21:12.081087 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:21:18.148797 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:21:18.148768 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbx5wbv"] Jul 10 02:21:18.152934 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:21:18.152901 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbx5wbv" Jul 10 02:21:18.156377 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:21:18.156345 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 10 02:21:18.162310 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:21:18.162286 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbx5wbv"] Jul 10 02:21:18.175738 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:21:18.175701 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:21:18.301680 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:21:18.301644 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63dccc93-8fe7-443d-b84f-11d2c041c13b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbx5wbv\" (UID: \"63dccc93-8fe7-443d-b84f-11d2c041c13b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbx5wbv" Jul 10 02:21:18.301680 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:21:18.301687 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63dccc93-8fe7-443d-b84f-11d2c041c13b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbx5wbv\" (UID: \"63dccc93-8fe7-443d-b84f-11d2c041c13b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbx5wbv" Jul 10 02:21:18.301908 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:21:18.301714 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63dccc93-8fe7-443d-b84f-11d2c041c13b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbx5wbv\" (UID: \"63dccc93-8fe7-443d-b84f-11d2c041c13b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbx5wbv" Jul 10 02:21:18.301908 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:21:18.301756 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63dccc93-8fe7-443d-b84f-11d2c041c13b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbx5wbv\" (UID: \"63dccc93-8fe7-443d-b84f-11d2c041c13b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbx5wbv" Jul 10 02:21:18.301908 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:21:18.301785 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63dccc93-8fe7-443d-b84f-11d2c041c13b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbx5wbv\" (UID: \"63dccc93-8fe7-443d-b84f-11d2c041c13b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbx5wbv" Jul 10 02:21:18.301908 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:21:18.301808 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5t87\" (UniqueName: \"kubernetes.io/projected/63dccc93-8fe7-443d-b84f-11d2c041c13b-kube-api-access-s5t87\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbx5wbv\" (UID: \"63dccc93-8fe7-443d-b84f-11d2c041c13b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbx5wbv" Jul 10 02:21:18.403297 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:21:18.403202 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63dccc93-8fe7-443d-b84f-11d2c041c13b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbx5wbv\" (UID: \"63dccc93-8fe7-443d-b84f-11d2c041c13b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbx5wbv" Jul 10 02:21:18.403297 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:21:18.403240 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63dccc93-8fe7-443d-b84f-11d2c041c13b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbx5wbv\" (UID: \"63dccc93-8fe7-443d-b84f-11d2c041c13b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbx5wbv" Jul 10 02:21:18.403297 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:21:18.403264 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63dccc93-8fe7-443d-b84f-11d2c041c13b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbx5wbv\" (UID: \"63dccc93-8fe7-443d-b84f-11d2c041c13b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbx5wbv" Jul 10 02:21:18.403297 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:21:18.403283 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63dccc93-8fe7-443d-b84f-11d2c041c13b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbx5wbv\" (UID: \"63dccc93-8fe7-443d-b84f-11d2c041c13b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbx5wbv" Jul 10 02:21:18.403678 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:21:18.403311 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63dccc93-8fe7-443d-b84f-11d2c041c13b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbx5wbv\" (UID: \"63dccc93-8fe7-443d-b84f-11d2c041c13b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbx5wbv" Jul 10 02:21:18.403678 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:21:18.403335 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5t87\" (UniqueName: \"kubernetes.io/projected/63dccc93-8fe7-443d-b84f-11d2c041c13b-kube-api-access-s5t87\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbx5wbv\" (UID: \"63dccc93-8fe7-443d-b84f-11d2c041c13b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbx5wbv" Jul 10 02:21:18.403790 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:21:18.403721 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63dccc93-8fe7-443d-b84f-11d2c041c13b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbx5wbv\" (UID: \"63dccc93-8fe7-443d-b84f-11d2c041c13b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbx5wbv" Jul 10 02:21:18.403790 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:21:18.403747 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63dccc93-8fe7-443d-b84f-11d2c041c13b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbx5wbv\" (UID: \"63dccc93-8fe7-443d-b84f-11d2c041c13b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbx5wbv" Jul 10 02:21:18.403883 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:21:18.403804 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63dccc93-8fe7-443d-b84f-11d2c041c13b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbx5wbv\" (UID: \"63dccc93-8fe7-443d-b84f-11d2c041c13b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbx5wbv" Jul 10 02:21:18.405541 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:21:18.405521 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63dccc93-8fe7-443d-b84f-11d2c041c13b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbx5wbv\" (UID: \"63dccc93-8fe7-443d-b84f-11d2c041c13b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbx5wbv" Jul 10 02:21:18.405825 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:21:18.405807 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63dccc93-8fe7-443d-b84f-11d2c041c13b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbx5wbv\" (UID: \"63dccc93-8fe7-443d-b84f-11d2c041c13b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbx5wbv" Jul 10 02:21:18.414983 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:21:18.414958 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5t87\" (UniqueName: \"kubernetes.io/projected/63dccc93-8fe7-443d-b84f-11d2c041c13b-kube-api-access-s5t87\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbx5wbv\" (UID: \"63dccc93-8fe7-443d-b84f-11d2c041c13b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbx5wbv" Jul 10 02:21:18.465037 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:21:18.464992 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbx5wbv" Jul 10 02:21:18.601112 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:21:18.601088 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbx5wbv"] Jul 10 02:21:18.603024 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:21:18.602984 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63dccc93_8fe7_443d_b84f_11d2c041c13b.slice/crio-b28372b1feb5cd3311895b186eb3cff4b5d02ebfdbe58e00b8a8dfb11cab3676 WatchSource:0}: Error finding container b28372b1feb5cd3311895b186eb3cff4b5d02ebfdbe58e00b8a8dfb11cab3676: Status 404 returned error can't find the container with id b28372b1feb5cd3311895b186eb3cff4b5d02ebfdbe58e00b8a8dfb11cab3676 Jul 10 02:21:19.384218 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:21:19.384185 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbx5wbv" event={"ID":"63dccc93-8fe7-443d-b84f-11d2c041c13b","Type":"ContainerStarted","Data":"0468cdf5afe82a98fcf515a9a7f41197669307e2558728c8dfbf530c3e6ed70e"} Jul 10 02:21:19.384750 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:21:19.384243 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbx5wbv" event={"ID":"63dccc93-8fe7-443d-b84f-11d2c041c13b","Type":"ContainerStarted","Data":"b28372b1feb5cd3311895b186eb3cff4b5d02ebfdbe58e00b8a8dfb11cab3676"} Jul 10 02:21:19.415286 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:21:19.415231 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbx5wbv" podStartSLOduration=1.2575444519999999 podStartE2EDuration="1.415216681s" totalImagesPullingTime="157.672229ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:21:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:21:19.414182156 +0000 UTC m=+810.990459270" watchObservedRunningTime="2026-07-10 02:21:19.415216681 +0000 UTC m=+810.991493766" Jul 10 02:21:28.466091 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:21:28.466060 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbx5wbv" Jul 10 02:21:28.466485 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:21:28.466142 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbx5wbv" Jul 10 02:21:28.478936 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:21:28.478916 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbx5wbv" Jul 10 02:21:29.446724 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:21:29.446695 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbx5wbv" Jul 10 02:21:30.212522 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:21:30.212485 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:21:33.183608 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:21:33.183558 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:22:08.027672 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:22:08.027629 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:22:08.131774 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:22:08.131736 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:22:29.467533 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:22:29.467499 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:22:29.543835 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:22:29.543795 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:22:30.673171 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:22:30.673144 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:22:30.770088 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:22:30.770058 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:22:39.359554 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:22:39.359517 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:22:39.411475 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:22:39.411438 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:23:10.112977 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:23:10.112947 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:23:46.207871 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:23:46.207806 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:23:47.874047 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:23:47.874010 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:24:09.237467 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:24:09.237412 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:24:28.689890 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:24:28.689853 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:24:28.793309 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:24:28.793273 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:24:28.939202 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:24:28.939173 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:24:35.338729 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:24:35.338691 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:24:35.440232 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:24:35.440193 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:24:35.538222 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:24:35.538179 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:24:35.759880 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:24:35.759787 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:24:40.088843 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:24:40.088803 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:24:41.978222 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:24:41.978186 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:24:42.078221 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:24:42.078186 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:25:50.162113 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:25:50.162043 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:29:40.906906 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:29:40.906877 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:30:00.146829 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:30:00.146795 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727510-sg6q6"] Jul 10 02:30:00.150497 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:30:00.150473 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727510-sg6q6" Jul 10 02:30:00.152960 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:30:00.152936 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rppvw\"" Jul 10 02:30:00.158278 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:30:00.158070 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727510-sg6q6"] Jul 10 02:30:00.162299 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:30:00.162279 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727510-sg6q6" Jul 10 02:30:00.305950 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:30:00.305928 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727510-sg6q6"] Jul 10 02:30:00.308012 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:30:00.307985 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaba74ed4_e7e7_4266_a503_fe797ef3b2ec.slice/crio-179c36b0b8d63d8dc1b640ec80bfeb012ffda53b3ec3fe8b8f7f863200a5b1ec WatchSource:0}: Error finding container 179c36b0b8d63d8dc1b640ec80bfeb012ffda53b3ec3fe8b8f7f863200a5b1ec: Status 404 returned error can't find the container with id 179c36b0b8d63d8dc1b640ec80bfeb012ffda53b3ec3fe8b8f7f863200a5b1ec Jul 10 02:30:00.309868 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:30:00.309847 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 02:30:01.054807 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:30:01.054741 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727510-sg6q6" event={"ID":"aba74ed4-e7e7-4266-a503-fe797ef3b2ec","Type":"ContainerStarted","Data":"179c36b0b8d63d8dc1b640ec80bfeb012ffda53b3ec3fe8b8f7f863200a5b1ec"} Jul 10 02:30:03.067071 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:30:03.067037 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727510-sg6q6" event={"ID":"aba74ed4-e7e7-4266-a503-fe797ef3b2ec","Type":"ContainerStarted","Data":"cc8e9c416b9c49cabe5d81898d0999d41b7a37200e98e8f210c2d5c266b620e9"} Jul 10 02:30:03.084859 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:30:03.084811 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727510-sg6q6" podStartSLOduration=0.715115711 podStartE2EDuration="3.084796266s" totalImagesPullingTime="2.369680555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:30:03.082872957 +0000 UTC m=+1334.659150056" watchObservedRunningTime="2026-07-10 02:30:03.084796266 +0000 UTC m=+1334.661073398" Jul 10 02:30:04.073890 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:30:04.073857 2566 generic.go:356] "Generic (PLEG): container finished" podID="aba74ed4-e7e7-4266-a503-fe797ef3b2ec" containerID="cc8e9c416b9c49cabe5d81898d0999d41b7a37200e98e8f210c2d5c266b620e9" exitCode=0 Jul 10 02:30:04.074281 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:30:04.073945 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727510-sg6q6" event={"ID":"aba74ed4-e7e7-4266-a503-fe797ef3b2ec","Type":"ContainerDied","Data":"cc8e9c416b9c49cabe5d81898d0999d41b7a37200e98e8f210c2d5c266b620e9"} Jul 10 02:30:05.207275 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:30:05.207249 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727510-sg6q6" Jul 10 02:30:06.085063 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:30:06.085031 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727510-sg6q6" event={"ID":"aba74ed4-e7e7-4266-a503-fe797ef3b2ec","Type":"ContainerDied","Data":"179c36b0b8d63d8dc1b640ec80bfeb012ffda53b3ec3fe8b8f7f863200a5b1ec"} Jul 10 02:30:06.085063 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:30:06.085056 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727510-sg6q6" Jul 10 02:30:06.085301 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:30:06.085074 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="179c36b0b8d63d8dc1b640ec80bfeb012ffda53b3ec3fe8b8f7f863200a5b1ec" Jul 10 02:33:23.809460 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:33:23.809426 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:33:23.897326 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:33:23.897262 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:33:38.122915 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:33:38.122871 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:34:40.881590 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:34:40.881503 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:34:40.987788 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:34:40.987752 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:35:29.520549 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:35:29.520512 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:35:41.318355 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:35:41.318322 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:35:49.595395 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:35:49.595359 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:35:54.767769 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:35:54.767736 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:35:57.781189 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:35:57.781153 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:35:57.879686 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:35:57.879654 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:35:58.258990 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:35:58.258911 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:36:04.972969 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:36:04.972938 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:36:05.048262 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:36:05.048225 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:36:05.195918 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:36:05.195885 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:36:05.295676 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:36:05.295635 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:36:06.392592 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:36:06.392558 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:36:06.483778 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:36:06.483725 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:36:06.584566 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:36:06.584529 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:36:06.880742 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:36:06.880711 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:36:07.228871 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:36:07.228790 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:36:07.329317 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:36:07.329286 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:36:08.932892 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:36:08.932865 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:36:11.446280 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:36:11.446199 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a1b34cb-mx7wz"] Jul 10 02:36:11.446716 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:36:11.446698 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aba74ed4-e7e7-4266-a503-fe797ef3b2ec" containerName="cleanup" Jul 10 02:36:11.446716 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:36:11.446717 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="aba74ed4-e7e7-4266-a503-fe797ef3b2ec" containerName="cleanup" Jul 10 02:36:11.446846 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:36:11.446785 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="aba74ed4-e7e7-4266-a503-fe797ef3b2ec" containerName="cleanup" Jul 10 02:36:11.449828 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:36:11.449808 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a1b34cb-mx7wz" Jul 10 02:36:11.452342 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:36:11.452320 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 02:36:11.452437 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:36:11.452338 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rppvw\"" Jul 10 02:36:11.458292 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:36:11.458272 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a1b34cb-mx7wz"] Jul 10 02:36:11.549313 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:36:11.549283 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76006c24-84be-4224-8133-bcdbf8f21acd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0a1b34cb-mx7wz\" (UID: \"76006c24-84be-4224-8133-bcdbf8f21acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a1b34cb-mx7wz" Jul 10 02:36:11.649707 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:36:11.649669 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76006c24-84be-4224-8133-bcdbf8f21acd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0a1b34cb-mx7wz\" (UID: \"76006c24-84be-4224-8133-bcdbf8f21acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a1b34cb-mx7wz" Jul 10 02:36:11.650268 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:36:11.650249 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76006c24-84be-4224-8133-bcdbf8f21acd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0a1b34cb-mx7wz\" (UID: \"76006c24-84be-4224-8133-bcdbf8f21acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a1b34cb-mx7wz" Jul 10 02:36:11.760283 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:36:11.760208 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a1b34cb-mx7wz" Jul 10 02:36:11.895308 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:36:11.895286 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a1b34cb-mx7wz"] Jul 10 02:36:11.897273 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:36:11.897243 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76006c24_84be_4224_8133_bcdbf8f21acd.slice/crio-3259316417d5ae00a0804b1d93b49b15b2b2236994f8dd729fd56fe900d11ac6 WatchSource:0}: Error finding container 3259316417d5ae00a0804b1d93b49b15b2b2236994f8dd729fd56fe900d11ac6: Status 404 returned error can't find the container with id 3259316417d5ae00a0804b1d93b49b15b2b2236994f8dd729fd56fe900d11ac6 Jul 10 02:36:12.016528 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:36:12.016472 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a1b34cb-mx7wz" event={"ID":"76006c24-84be-4224-8133-bcdbf8f21acd","Type":"ContainerStarted","Data":"3259316417d5ae00a0804b1d93b49b15b2b2236994f8dd729fd56fe900d11ac6"} Jul 10 02:36:13.022672 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:36:13.022638 2566 generic.go:356] "Generic (PLEG): container finished" podID="76006c24-84be-4224-8133-bcdbf8f21acd" containerID="b1b218d60651af61d82d9f3e83111a02c0ea4cd6ede31cc9ac8ae481c37b2e55" exitCode=7 Jul 10 02:36:13.023056 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:36:13.022690 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a1b34cb-mx7wz" event={"ID":"76006c24-84be-4224-8133-bcdbf8f21acd","Type":"ContainerDied","Data":"b1b218d60651af61d82d9f3e83111a02c0ea4cd6ede31cc9ac8ae481c37b2e55"} Jul 10 02:36:13.023056 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:36:13.022846 2566 scope.go:117] "RemoveContainer" containerID="b1b218d60651af61d82d9f3e83111a02c0ea4cd6ede31cc9ac8ae481c37b2e55" Jul 10 02:36:14.030002 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:36:14.029964 2566 generic.go:356] "Generic (PLEG): container finished" podID="76006c24-84be-4224-8133-bcdbf8f21acd" containerID="a52ac80c5d32a4d0a1636edd344cf1be281bb4633f7eb96a815bf5622cdfdbfe" exitCode=7 Jul 10 02:36:14.030483 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:36:14.030057 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a1b34cb-mx7wz" event={"ID":"76006c24-84be-4224-8133-bcdbf8f21acd","Type":"ContainerDied","Data":"a52ac80c5d32a4d0a1636edd344cf1be281bb4633f7eb96a815bf5622cdfdbfe"} Jul 10 02:36:14.030483 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:36:14.030180 2566 scope.go:117] "RemoveContainer" containerID="b1b218d60651af61d82d9f3e83111a02c0ea4cd6ede31cc9ac8ae481c37b2e55" Jul 10 02:36:14.030483 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:36:14.030444 2566 scope.go:117] "RemoveContainer" containerID="a52ac80c5d32a4d0a1636edd344cf1be281bb4633f7eb96a815bf5622cdfdbfe" Jul 10 02:36:14.030693 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:36:14.030675 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0a1b34cb-mx7wz_odh-ai-gateway-infra(76006c24-84be-4224-8133-bcdbf8f21acd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a1b34cb-mx7wz" podUID="76006c24-84be-4224-8133-bcdbf8f21acd" Jul 10 02:36:15.036901 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:36:15.036876 2566 scope.go:117] "RemoveContainer" containerID="a52ac80c5d32a4d0a1636edd344cf1be281bb4633f7eb96a815bf5622cdfdbfe" Jul 10 02:36:15.037257 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:36:15.037086 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0a1b34cb-mx7wz_odh-ai-gateway-infra(76006c24-84be-4224-8133-bcdbf8f21acd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a1b34cb-mx7wz" podUID="76006c24-84be-4224-8133-bcdbf8f21acd" Jul 10 02:36:27.005951 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:36:27.005919 2566 scope.go:117] "RemoveContainer" containerID="a52ac80c5d32a4d0a1636edd344cf1be281bb4633f7eb96a815bf5622cdfdbfe" Jul 10 02:36:28.106761 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:36:28.106726 2566 generic.go:356] "Generic (PLEG): container finished" podID="76006c24-84be-4224-8133-bcdbf8f21acd" containerID="65529094217b56135a1948a0b3a3bf2ae660c662b2a9c21dc23cc4397e76d1f2" exitCode=0 Jul 10 02:36:28.107185 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:36:28.106783 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a1b34cb-mx7wz" event={"ID":"76006c24-84be-4224-8133-bcdbf8f21acd","Type":"ContainerDied","Data":"65529094217b56135a1948a0b3a3bf2ae660c662b2a9c21dc23cc4397e76d1f2"} Jul 10 02:36:28.107185 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:36:28.106830 2566 scope.go:117] "RemoveContainer" containerID="a52ac80c5d32a4d0a1636edd344cf1be281bb4633f7eb96a815bf5622cdfdbfe" Jul 10 02:36:29.141885 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:36:29.141850 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a1b34cb-mx7wz"] Jul 10 02:36:29.245636 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:36:29.245592 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a1b34cb-mx7wz" Jul 10 02:36:29.292672 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:36:29.292646 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76006c24-84be-4224-8133-bcdbf8f21acd-maas-api-service-ca\") pod \"76006c24-84be-4224-8133-bcdbf8f21acd\" (UID: \"76006c24-84be-4224-8133-bcdbf8f21acd\") " Jul 10 02:36:29.292976 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:36:29.292952 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76006c24-84be-4224-8133-bcdbf8f21acd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "76006c24-84be-4224-8133-bcdbf8f21acd" (UID: "76006c24-84be-4224-8133-bcdbf8f21acd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 02:36:29.393981 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:36:29.393923 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76006c24-84be-4224-8133-bcdbf8f21acd-maas-api-service-ca\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:36:30.119367 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:36:30.119332 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a1b34cb-mx7wz" event={"ID":"76006c24-84be-4224-8133-bcdbf8f21acd","Type":"ContainerDied","Data":"3259316417d5ae00a0804b1d93b49b15b2b2236994f8dd729fd56fe900d11ac6"} Jul 10 02:36:30.119367 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:36:30.119354 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a1b34cb-mx7wz" Jul 10 02:36:30.119588 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:36:30.119382 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3259316417d5ae00a0804b1d93b49b15b2b2236994f8dd729fd56fe900d11ac6" Jul 10 02:36:30.153366 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:36:30.153334 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a1b34cb-mx7wz"] Jul 10 02:36:30.158419 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:36:30.158390 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a1b34cb-mx7wz"] Jul 10 02:36:31.011039 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:36:31.011007 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76006c24-84be-4224-8133-bcdbf8f21acd" path="/var/lib/kubelet/pods/76006c24-84be-4224-8133-bcdbf8f21acd/volumes" Jul 10 02:36:45.237892 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:36:45.237848 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:37:01.506860 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:01.506826 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a1b34cb-ct4fp"] Jul 10 02:37:01.507321 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:01.507161 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76006c24-84be-4224-8133-bcdbf8f21acd" containerName="revoke-keys" Jul 10 02:37:01.507321 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:01.507171 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="76006c24-84be-4224-8133-bcdbf8f21acd" containerName="revoke-keys" Jul 10 02:37:01.507321 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:01.507187 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76006c24-84be-4224-8133-bcdbf8f21acd" containerName="revoke-keys" Jul 10 02:37:01.507321 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:01.507192 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="76006c24-84be-4224-8133-bcdbf8f21acd" containerName="revoke-keys" Jul 10 02:37:01.507321 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:01.507260 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="76006c24-84be-4224-8133-bcdbf8f21acd" containerName="revoke-keys" Jul 10 02:37:01.507321 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:01.507270 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="76006c24-84be-4224-8133-bcdbf8f21acd" containerName="revoke-keys" Jul 10 02:37:01.510274 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:01.510255 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a1b34cb-ct4fp" Jul 10 02:37:01.512770 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:01.512748 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rppvw\"" Jul 10 02:37:01.512989 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:01.512977 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 02:37:01.518345 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:01.518319 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a1b34cb-ct4fp"] Jul 10 02:37:01.663871 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:01.663843 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8fbd472-02cc-43e1-97a2-0c0e3b64c734-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0a1b34cb-ct4fp\" (UID: \"f8fbd472-02cc-43e1-97a2-0c0e3b64c734\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a1b34cb-ct4fp" Jul 10 02:37:01.765271 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:01.765207 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8fbd472-02cc-43e1-97a2-0c0e3b64c734-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0a1b34cb-ct4fp\" (UID: \"f8fbd472-02cc-43e1-97a2-0c0e3b64c734\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a1b34cb-ct4fp" Jul 10 02:37:01.765799 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:01.765782 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8fbd472-02cc-43e1-97a2-0c0e3b64c734-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0a1b34cb-ct4fp\" (UID: \"f8fbd472-02cc-43e1-97a2-0c0e3b64c734\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a1b34cb-ct4fp" Jul 10 02:37:01.821958 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:01.821937 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a1b34cb-ct4fp" Jul 10 02:37:01.960077 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:01.960054 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a1b34cb-ct4fp"] Jul 10 02:37:01.962405 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:37:01.962380 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8fbd472_02cc_43e1_97a2_0c0e3b64c734.slice/crio-21237e3e5e7343a249018c3ae626b6d7bba5919a7c877c219a626542ac19aa6e WatchSource:0}: Error finding container 21237e3e5e7343a249018c3ae626b6d7bba5919a7c877c219a626542ac19aa6e: Status 404 returned error can't find the container with id 21237e3e5e7343a249018c3ae626b6d7bba5919a7c877c219a626542ac19aa6e Jul 10 02:37:02.291963 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:02.291934 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a1b34cb-ct4fp" event={"ID":"f8fbd472-02cc-43e1-97a2-0c0e3b64c734","Type":"ContainerStarted","Data":"21237e3e5e7343a249018c3ae626b6d7bba5919a7c877c219a626542ac19aa6e"} Jul 10 02:37:03.298401 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:03.298363 2566 generic.go:356] "Generic (PLEG): container finished" podID="f8fbd472-02cc-43e1-97a2-0c0e3b64c734" containerID="1284d95c9a49714448162630d7148f7cc41ca9bc621e75a6c54547187133c0d3" exitCode=0 Jul 10 02:37:03.298814 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:03.298412 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a1b34cb-ct4fp" event={"ID":"f8fbd472-02cc-43e1-97a2-0c0e3b64c734","Type":"ContainerDied","Data":"1284d95c9a49714448162630d7148f7cc41ca9bc621e75a6c54547187133c0d3"} Jul 10 02:37:04.444835 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:04.444814 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a1b34cb-ct4fp" Jul 10 02:37:04.587971 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:04.587881 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8fbd472-02cc-43e1-97a2-0c0e3b64c734-maas-api-service-ca\") pod \"f8fbd472-02cc-43e1-97a2-0c0e3b64c734\" (UID: \"f8fbd472-02cc-43e1-97a2-0c0e3b64c734\") " Jul 10 02:37:04.588227 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:04.588205 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8fbd472-02cc-43e1-97a2-0c0e3b64c734-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f8fbd472-02cc-43e1-97a2-0c0e3b64c734" (UID: "f8fbd472-02cc-43e1-97a2-0c0e3b64c734"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 02:37:04.689372 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:04.689329 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8fbd472-02cc-43e1-97a2-0c0e3b64c734-maas-api-service-ca\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:37:05.309744 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:05.309663 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a1b34cb-ct4fp" Jul 10 02:37:05.309744 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:05.309686 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a1b34cb-ct4fp" event={"ID":"f8fbd472-02cc-43e1-97a2-0c0e3b64c734","Type":"ContainerDied","Data":"21237e3e5e7343a249018c3ae626b6d7bba5919a7c877c219a626542ac19aa6e"} Jul 10 02:37:05.309744 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:05.309733 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21237e3e5e7343a249018c3ae626b6d7bba5919a7c877c219a626542ac19aa6e" Jul 10 02:37:11.583372 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:11.583341 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:37:11.704974 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:11.704941 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:37:12.065051 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:12.065016 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:37:18.092030 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:18.092001 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:37:18.255382 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:18.255350 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:37:18.357886 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:18.357801 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:37:18.470668 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:18.470635 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:37:18.546665 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:18.546628 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:37:19.585906 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:19.585864 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:37:19.672485 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:19.672457 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:37:19.746844 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:19.746812 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:37:19.900119 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:19.900050 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:37:20.311187 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:20.311145 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:37:20.411184 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:20.411156 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:37:22.107287 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:22.107257 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:37:24.733690 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:24.733656 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc69ef18-vvkgf"] Jul 10 02:37:24.734204 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:24.734185 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8fbd472-02cc-43e1-97a2-0c0e3b64c734" containerName="revoke-keys" Jul 10 02:37:24.734299 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:24.734207 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8fbd472-02cc-43e1-97a2-0c0e3b64c734" containerName="revoke-keys" Jul 10 02:37:24.734299 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:24.734246 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76006c24-84be-4224-8133-bcdbf8f21acd" containerName="revoke-keys" Jul 10 02:37:24.734299 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:24.734254 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="76006c24-84be-4224-8133-bcdbf8f21acd" containerName="revoke-keys" Jul 10 02:37:24.734455 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:24.734351 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8fbd472-02cc-43e1-97a2-0c0e3b64c734" containerName="revoke-keys" Jul 10 02:37:24.734455 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:24.734369 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="76006c24-84be-4224-8133-bcdbf8f21acd" containerName="revoke-keys" Jul 10 02:37:24.737420 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:24.737400 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc69ef18-vvkgf" Jul 10 02:37:24.739936 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:24.739916 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rppvw\"" Jul 10 02:37:24.739936 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:24.739933 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 02:37:24.745849 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:24.745828 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc69ef18-vvkgf"] Jul 10 02:37:24.748494 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:24.748467 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3454db3b-52c8-400b-a2a5-a6c4bcdef55f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-bc69ef18-vvkgf\" (UID: \"3454db3b-52c8-400b-a2a5-a6c4bcdef55f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc69ef18-vvkgf" Jul 10 02:37:24.849073 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:24.849044 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3454db3b-52c8-400b-a2a5-a6c4bcdef55f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-bc69ef18-vvkgf\" (UID: \"3454db3b-52c8-400b-a2a5-a6c4bcdef55f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc69ef18-vvkgf" Jul 10 02:37:24.849697 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:24.849678 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3454db3b-52c8-400b-a2a5-a6c4bcdef55f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-bc69ef18-vvkgf\" (UID: \"3454db3b-52c8-400b-a2a5-a6c4bcdef55f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc69ef18-vvkgf" Jul 10 02:37:25.048822 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:25.048794 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc69ef18-vvkgf" Jul 10 02:37:25.181374 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:25.181339 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc69ef18-vvkgf"] Jul 10 02:37:25.185087 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:37:25.185058 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3454db3b_52c8_400b_a2a5_a6c4bcdef55f.slice/crio-a2d4c3d6a792ceb2da8dd97710c528df354527300ba0673941a7c12825c96a1f WatchSource:0}: Error finding container a2d4c3d6a792ceb2da8dd97710c528df354527300ba0673941a7c12825c96a1f: Status 404 returned error can't find the container with id a2d4c3d6a792ceb2da8dd97710c528df354527300ba0673941a7c12825c96a1f Jul 10 02:37:25.419387 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:25.419345 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc69ef18-vvkgf" event={"ID":"3454db3b-52c8-400b-a2a5-a6c4bcdef55f","Type":"ContainerStarted","Data":"a2d4c3d6a792ceb2da8dd97710c528df354527300ba0673941a7c12825c96a1f"} Jul 10 02:37:26.425725 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:26.425691 2566 generic.go:356] "Generic (PLEG): container finished" podID="3454db3b-52c8-400b-a2a5-a6c4bcdef55f" containerID="d52b3e88a69e05ed39f72d2791a5e4a3f688d1516254c1f6be491a29b20e2556" exitCode=7 Jul 10 02:37:26.426141 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:26.425768 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc69ef18-vvkgf" event={"ID":"3454db3b-52c8-400b-a2a5-a6c4bcdef55f","Type":"ContainerDied","Data":"d52b3e88a69e05ed39f72d2791a5e4a3f688d1516254c1f6be491a29b20e2556"} Jul 10 02:37:26.426141 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:26.425946 2566 scope.go:117] "RemoveContainer" containerID="d52b3e88a69e05ed39f72d2791a5e4a3f688d1516254c1f6be491a29b20e2556" Jul 10 02:37:27.434737 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:27.434655 2566 generic.go:356] "Generic (PLEG): container finished" podID="3454db3b-52c8-400b-a2a5-a6c4bcdef55f" containerID="85511f6746271511f34b64f2f6182fc926a15306ed75b62cfc6bca9d1be436bb" exitCode=7 Jul 10 02:37:27.435119 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:27.434743 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc69ef18-vvkgf" event={"ID":"3454db3b-52c8-400b-a2a5-a6c4bcdef55f","Type":"ContainerDied","Data":"85511f6746271511f34b64f2f6182fc926a15306ed75b62cfc6bca9d1be436bb"} Jul 10 02:37:27.435119 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:27.434829 2566 scope.go:117] "RemoveContainer" containerID="d52b3e88a69e05ed39f72d2791a5e4a3f688d1516254c1f6be491a29b20e2556" Jul 10 02:37:27.435119 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:27.435073 2566 scope.go:117] "RemoveContainer" containerID="85511f6746271511f34b64f2f6182fc926a15306ed75b62cfc6bca9d1be436bb" Jul 10 02:37:27.435298 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:37:27.435277 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-bc69ef18-vvkgf_odh-ai-gateway-infra(3454db3b-52c8-400b-a2a5-a6c4bcdef55f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc69ef18-vvkgf" podUID="3454db3b-52c8-400b-a2a5-a6c4bcdef55f" Jul 10 02:37:28.441277 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:28.441250 2566 scope.go:117] "RemoveContainer" containerID="85511f6746271511f34b64f2f6182fc926a15306ed75b62cfc6bca9d1be436bb" Jul 10 02:37:28.441670 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:37:28.441426 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-bc69ef18-vvkgf_odh-ai-gateway-infra(3454db3b-52c8-400b-a2a5-a6c4bcdef55f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc69ef18-vvkgf" podUID="3454db3b-52c8-400b-a2a5-a6c4bcdef55f" Jul 10 02:37:43.005330 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:43.005255 2566 scope.go:117] "RemoveContainer" containerID="85511f6746271511f34b64f2f6182fc926a15306ed75b62cfc6bca9d1be436bb" Jul 10 02:37:43.528982 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:43.528943 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc69ef18-vvkgf" event={"ID":"3454db3b-52c8-400b-a2a5-a6c4bcdef55f","Type":"ContainerStarted","Data":"bfbd0e6f7d27032eb53b55f494c17df68126e40ec2512df16717d7164d382f38"} Jul 10 02:37:43.547947 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:43.547891 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc69ef18-vvkgf" podStartSLOduration=19.547876465 podStartE2EDuration="19.547876465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:37:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:37:43.545376936 +0000 UTC m=+1795.121654021" watchObservedRunningTime="2026-07-10 02:37:43.547876465 +0000 UTC m=+1795.124153551" Jul 10 02:37:44.535924 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:44.535888 2566 generic.go:356] "Generic (PLEG): container finished" podID="3454db3b-52c8-400b-a2a5-a6c4bcdef55f" containerID="bfbd0e6f7d27032eb53b55f494c17df68126e40ec2512df16717d7164d382f38" exitCode=0 Jul 10 02:37:44.536325 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:44.535944 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc69ef18-vvkgf" event={"ID":"3454db3b-52c8-400b-a2a5-a6c4bcdef55f","Type":"ContainerDied","Data":"bfbd0e6f7d27032eb53b55f494c17df68126e40ec2512df16717d7164d382f38"} Jul 10 02:37:44.536325 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:44.535990 2566 scope.go:117] "RemoveContainer" containerID="85511f6746271511f34b64f2f6182fc926a15306ed75b62cfc6bca9d1be436bb" Jul 10 02:37:44.565555 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:44.565519 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc69ef18-vvkgf"] Jul 10 02:37:45.680692 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:45.680665 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc69ef18-vvkgf" Jul 10 02:37:45.828375 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:45.828288 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3454db3b-52c8-400b-a2a5-a6c4bcdef55f-maas-api-service-ca\") pod \"3454db3b-52c8-400b-a2a5-a6c4bcdef55f\" (UID: \"3454db3b-52c8-400b-a2a5-a6c4bcdef55f\") " Jul 10 02:37:45.828657 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:45.828634 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3454db3b-52c8-400b-a2a5-a6c4bcdef55f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3454db3b-52c8-400b-a2a5-a6c4bcdef55f" (UID: "3454db3b-52c8-400b-a2a5-a6c4bcdef55f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 02:37:45.929489 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:45.929461 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3454db3b-52c8-400b-a2a5-a6c4bcdef55f-maas-api-service-ca\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:37:46.551966 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:46.551921 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc69ef18-vvkgf" event={"ID":"3454db3b-52c8-400b-a2a5-a6c4bcdef55f","Type":"ContainerDied","Data":"a2d4c3d6a792ceb2da8dd97710c528df354527300ba0673941a7c12825c96a1f"} Jul 10 02:37:46.552116 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:46.552005 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2d4c3d6a792ceb2da8dd97710c528df354527300ba0673941a7c12825c96a1f" Jul 10 02:37:46.552116 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:46.551935 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc69ef18-vvkgf" Jul 10 02:37:46.588224 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:46.588194 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc69ef18-vvkgf"] Jul 10 02:37:46.590526 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:46.590499 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc69ef18-vvkgf"] Jul 10 02:37:47.011711 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:37:47.011679 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3454db3b-52c8-400b-a2a5-a6c4bcdef55f" path="/var/lib/kubelet/pods/3454db3b-52c8-400b-a2a5-a6c4bcdef55f/volumes" Jul 10 02:38:24.794817 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:24.794785 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc69ef18-r8mp9"] Jul 10 02:38:24.795245 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:24.795126 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3454db3b-52c8-400b-a2a5-a6c4bcdef55f" containerName="revoke-keys" Jul 10 02:38:24.795245 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:24.795138 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3454db3b-52c8-400b-a2a5-a6c4bcdef55f" containerName="revoke-keys" Jul 10 02:38:24.795245 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:24.795150 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3454db3b-52c8-400b-a2a5-a6c4bcdef55f" containerName="revoke-keys" Jul 10 02:38:24.795245 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:24.795157 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3454db3b-52c8-400b-a2a5-a6c4bcdef55f" containerName="revoke-keys" Jul 10 02:38:24.795245 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:24.795171 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3454db3b-52c8-400b-a2a5-a6c4bcdef55f" containerName="revoke-keys" Jul 10 02:38:24.795245 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:24.795177 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3454db3b-52c8-400b-a2a5-a6c4bcdef55f" containerName="revoke-keys" Jul 10 02:38:24.795245 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:24.795236 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3454db3b-52c8-400b-a2a5-a6c4bcdef55f" containerName="revoke-keys" Jul 10 02:38:24.795245 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:24.795244 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3454db3b-52c8-400b-a2a5-a6c4bcdef55f" containerName="revoke-keys" Jul 10 02:38:24.798385 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:24.798367 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc69ef18-r8mp9" Jul 10 02:38:24.800928 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:24.800906 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 02:38:24.801247 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:24.801228 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rppvw\"" Jul 10 02:38:24.807063 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:24.807044 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc69ef18-r8mp9"] Jul 10 02:38:24.859665 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:24.859636 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0350ca0-9993-4f12-ba84-93164b133483-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-bc69ef18-r8mp9\" (UID: \"d0350ca0-9993-4f12-ba84-93164b133483\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc69ef18-r8mp9" Jul 10 02:38:24.960560 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:24.960528 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0350ca0-9993-4f12-ba84-93164b133483-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-bc69ef18-r8mp9\" (UID: \"d0350ca0-9993-4f12-ba84-93164b133483\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc69ef18-r8mp9" Jul 10 02:38:24.961119 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:24.961098 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0350ca0-9993-4f12-ba84-93164b133483-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-bc69ef18-r8mp9\" (UID: \"d0350ca0-9993-4f12-ba84-93164b133483\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc69ef18-r8mp9" Jul 10 02:38:25.109195 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:25.109113 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc69ef18-r8mp9" Jul 10 02:38:25.244738 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:25.244712 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc69ef18-r8mp9"] Jul 10 02:38:25.759493 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:25.759404 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc69ef18-r8mp9" event={"ID":"d0350ca0-9993-4f12-ba84-93164b133483","Type":"ContainerStarted","Data":"43ec6dcb3305ec8324b6cf1def89384e2770111a32e924db71dd16d2096890b2"} Jul 10 02:38:25.759493 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:25.759466 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc69ef18-r8mp9" event={"ID":"d0350ca0-9993-4f12-ba84-93164b133483","Type":"ContainerStarted","Data":"e96a9e123e721e0d3cc8cb29f8f80af14a90d2e264d95acac61915e4e52f0d33"} Jul 10 02:38:25.780102 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:25.780054 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc69ef18-r8mp9" podStartSLOduration=1.780038114 podStartE2EDuration="1.780038114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:38:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:38:25.777139308 +0000 UTC m=+1837.353416393" watchObservedRunningTime="2026-07-10 02:38:25.780038114 +0000 UTC m=+1837.356315208" Jul 10 02:38:26.766425 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:26.766386 2566 generic.go:356] "Generic (PLEG): container finished" podID="d0350ca0-9993-4f12-ba84-93164b133483" containerID="43ec6dcb3305ec8324b6cf1def89384e2770111a32e924db71dd16d2096890b2" exitCode=0 Jul 10 02:38:26.766831 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:26.766473 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc69ef18-r8mp9" event={"ID":"d0350ca0-9993-4f12-ba84-93164b133483","Type":"ContainerDied","Data":"43ec6dcb3305ec8324b6cf1def89384e2770111a32e924db71dd16d2096890b2"} Jul 10 02:38:27.903857 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:27.903826 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc69ef18-r8mp9" Jul 10 02:38:27.987519 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:27.987488 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0350ca0-9993-4f12-ba84-93164b133483-maas-api-service-ca\") pod \"d0350ca0-9993-4f12-ba84-93164b133483\" (UID: \"d0350ca0-9993-4f12-ba84-93164b133483\") " Jul 10 02:38:27.987842 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:27.987817 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0350ca0-9993-4f12-ba84-93164b133483-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d0350ca0-9993-4f12-ba84-93164b133483" (UID: "d0350ca0-9993-4f12-ba84-93164b133483"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 02:38:28.088831 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:28.088743 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0350ca0-9993-4f12-ba84-93164b133483-maas-api-service-ca\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:38:28.778713 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:28.778681 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc69ef18-r8mp9" event={"ID":"d0350ca0-9993-4f12-ba84-93164b133483","Type":"ContainerDied","Data":"e96a9e123e721e0d3cc8cb29f8f80af14a90d2e264d95acac61915e4e52f0d33"} Jul 10 02:38:28.778713 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:28.778697 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc69ef18-r8mp9" Jul 10 02:38:28.778917 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:28.778733 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e96a9e123e721e0d3cc8cb29f8f80af14a90d2e264d95acac61915e4e52f0d33" Jul 10 02:38:35.352463 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:35.352425 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:38:39.902772 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:39.902730 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:38:46.183853 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:46.183813 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:38:46.241048 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:46.241021 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:38:50.135194 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:50.135148 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:38:50.539495 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:50.539464 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:38:51.651829 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:51.651794 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:38:51.728142 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:51.728113 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:38:51.733178 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:51.733147 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-987cdfe4-67bf7fb44b-vhj2l"] Jul 10 02:38:51.733674 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:51.733655 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0350ca0-9993-4f12-ba84-93164b133483" containerName="revoke-keys" Jul 10 02:38:51.733674 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:51.733673 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0350ca0-9993-4f12-ba84-93164b133483" containerName="revoke-keys" Jul 10 02:38:51.733832 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:51.733744 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0350ca0-9993-4f12-ba84-93164b133483" containerName="revoke-keys" Jul 10 02:38:51.733832 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:51.733754 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3454db3b-52c8-400b-a2a5-a6c4bcdef55f" containerName="revoke-keys" Jul 10 02:38:51.736644 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:51.736627 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-987cdfe4-67bf7fb44b-vhj2l" Jul 10 02:38:51.739656 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:51.739634 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 10 02:38:51.739767 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:51.739676 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 02:38:51.740895 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:51.740875 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-987cdfe4-serving-cert\"" Jul 10 02:38:51.741007 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:51.740897 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-r7s9m\"" Jul 10 02:38:51.747142 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:51.747119 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-987cdfe4-67bf7fb44b-vhj2l"] Jul 10 02:38:51.810074 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:51.810049 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:38:51.891769 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:51.891733 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rfh9\" (UniqueName: \"kubernetes.io/projected/eed197ab-a025-42dc-a504-05ecb9612957-kube-api-access-5rfh9\") pod \"maas-api-e2e-derive-987cdfe4-67bf7fb44b-vhj2l\" (UID: \"eed197ab-a025-42dc-a504-05ecb9612957\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-987cdfe4-67bf7fb44b-vhj2l" Jul 10 02:38:51.891913 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:51.891772 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eed197ab-a025-42dc-a504-05ecb9612957-openshift-service-ca\") pod \"maas-api-e2e-derive-987cdfe4-67bf7fb44b-vhj2l\" (UID: \"eed197ab-a025-42dc-a504-05ecb9612957\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-987cdfe4-67bf7fb44b-vhj2l" Jul 10 02:38:51.891913 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:51.891834 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eed197ab-a025-42dc-a504-05ecb9612957-maas-api-tls\") pod \"maas-api-e2e-derive-987cdfe4-67bf7fb44b-vhj2l\" (UID: \"eed197ab-a025-42dc-a504-05ecb9612957\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-987cdfe4-67bf7fb44b-vhj2l" Jul 10 02:38:51.992453 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:51.992388 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rfh9\" (UniqueName: \"kubernetes.io/projected/eed197ab-a025-42dc-a504-05ecb9612957-kube-api-access-5rfh9\") pod \"maas-api-e2e-derive-987cdfe4-67bf7fb44b-vhj2l\" (UID: \"eed197ab-a025-42dc-a504-05ecb9612957\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-987cdfe4-67bf7fb44b-vhj2l" Jul 10 02:38:51.992453 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:51.992417 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eed197ab-a025-42dc-a504-05ecb9612957-openshift-service-ca\") pod \"maas-api-e2e-derive-987cdfe4-67bf7fb44b-vhj2l\" (UID: \"eed197ab-a025-42dc-a504-05ecb9612957\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-987cdfe4-67bf7fb44b-vhj2l" Jul 10 02:38:51.992453 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:51.992450 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eed197ab-a025-42dc-a504-05ecb9612957-maas-api-tls\") pod \"maas-api-e2e-derive-987cdfe4-67bf7fb44b-vhj2l\" (UID: \"eed197ab-a025-42dc-a504-05ecb9612957\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-987cdfe4-67bf7fb44b-vhj2l" Jul 10 02:38:51.992762 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:38:51.992575 2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-987cdfe4-serving-cert: secret "maas-api-e2e-derive-987cdfe4-serving-cert" not found Jul 10 02:38:51.992762 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:38:51.992652 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eed197ab-a025-42dc-a504-05ecb9612957-maas-api-tls podName:eed197ab-a025-42dc-a504-05ecb9612957 nodeName:}" failed. No retries permitted until 2026-07-10 02:38:52.492631454 +0000 UTC m=+1864.068908527 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eed197ab-a025-42dc-a504-05ecb9612957-maas-api-tls") pod "maas-api-e2e-derive-987cdfe4-67bf7fb44b-vhj2l" (UID: "eed197ab-a025-42dc-a504-05ecb9612957") : secret "maas-api-e2e-derive-987cdfe4-serving-cert" not found Jul 10 02:38:51.993122 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:51.993100 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eed197ab-a025-42dc-a504-05ecb9612957-openshift-service-ca\") pod \"maas-api-e2e-derive-987cdfe4-67bf7fb44b-vhj2l\" (UID: \"eed197ab-a025-42dc-a504-05ecb9612957\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-987cdfe4-67bf7fb44b-vhj2l" Jul 10 02:38:52.004415 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:52.004393 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rfh9\" (UniqueName: \"kubernetes.io/projected/eed197ab-a025-42dc-a504-05ecb9612957-kube-api-access-5rfh9\") pod \"maas-api-e2e-derive-987cdfe4-67bf7fb44b-vhj2l\" (UID: \"eed197ab-a025-42dc-a504-05ecb9612957\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-987cdfe4-67bf7fb44b-vhj2l" Jul 10 02:38:52.231991 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:52.231955 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:38:52.426570 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:52.426542 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:38:52.496877 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:52.496851 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eed197ab-a025-42dc-a504-05ecb9612957-maas-api-tls\") pod \"maas-api-e2e-derive-987cdfe4-67bf7fb44b-vhj2l\" (UID: \"eed197ab-a025-42dc-a504-05ecb9612957\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-987cdfe4-67bf7fb44b-vhj2l" Jul 10 02:38:52.499203 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:52.499176 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eed197ab-a025-42dc-a504-05ecb9612957-maas-api-tls\") pod \"maas-api-e2e-derive-987cdfe4-67bf7fb44b-vhj2l\" (UID: \"eed197ab-a025-42dc-a504-05ecb9612957\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-987cdfe4-67bf7fb44b-vhj2l" Jul 10 02:38:52.528366 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:52.528339 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:38:52.650415 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:52.650388 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-987cdfe4-67bf7fb44b-vhj2l" Jul 10 02:38:52.787457 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:52.787434 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-987cdfe4-67bf7fb44b-vhj2l"] Jul 10 02:38:52.789346 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:38:52.789319 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeed197ab_a025_42dc_a504_05ecb9612957.slice/crio-de9fec17864702b8a51a34ffae9df01bf1c83ac99deed85de80231cc71897516 WatchSource:0}: Error finding container de9fec17864702b8a51a34ffae9df01bf1c83ac99deed85de80231cc71897516: Status 404 returned error can't find the container with id de9fec17864702b8a51a34ffae9df01bf1c83ac99deed85de80231cc71897516 Jul 10 02:38:52.790826 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:52.790809 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 02:38:52.906685 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:52.906627 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-987cdfe4-67bf7fb44b-vhj2l" event={"ID":"eed197ab-a025-42dc-a504-05ecb9612957","Type":"ContainerStarted","Data":"de9fec17864702b8a51a34ffae9df01bf1c83ac99deed85de80231cc71897516"} Jul 10 02:38:54.920374 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:54.920343 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-987cdfe4-67bf7fb44b-vhj2l" event={"ID":"eed197ab-a025-42dc-a504-05ecb9612957","Type":"ContainerStarted","Data":"393ddef7d5e8777cdbee94b556cb6f3cd4d7fec12e8b4f6ffdc3784ec633a5dd"} Jul 10 02:38:54.920761 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:54.920413 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-987cdfe4-67bf7fb44b-vhj2l" Jul 10 02:38:54.940295 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:54.940247 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-987cdfe4-67bf7fb44b-vhj2l" podStartSLOduration=2.28489113 podStartE2EDuration="3.940232295s" totalImagesPullingTime="1.655341165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:38:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:38:54.938346527 +0000 UTC m=+1866.514623637" watchObservedRunningTime="2026-07-10 02:38:54.940232295 +0000 UTC m=+1866.516509381" Jul 10 02:38:56.077486 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:56.077452 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-987cdfe4-79q66"] Jul 10 02:38:56.080965 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:56.080943 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-987cdfe4-79q66" Jul 10 02:38:56.083433 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:56.083412 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rppvw\"" Jul 10 02:38:56.088039 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:56.088014 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-987cdfe4-79q66"] Jul 10 02:38:56.228294 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:56.228264 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3859c55-754e-41cf-9ac5-e755a6d94e6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-987cdfe4-79q66\" (UID: \"d3859c55-754e-41cf-9ac5-e755a6d94e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-987cdfe4-79q66" Jul 10 02:38:56.328925 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:56.328854 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3859c55-754e-41cf-9ac5-e755a6d94e6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-987cdfe4-79q66\" (UID: \"d3859c55-754e-41cf-9ac5-e755a6d94e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-987cdfe4-79q66" Jul 10 02:38:56.329473 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:56.329455 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3859c55-754e-41cf-9ac5-e755a6d94e6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-987cdfe4-79q66\" (UID: \"d3859c55-754e-41cf-9ac5-e755a6d94e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-987cdfe4-79q66" Jul 10 02:38:56.391967 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:56.391931 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-987cdfe4-79q66" Jul 10 02:38:56.521207 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:56.521180 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-987cdfe4-79q66"] Jul 10 02:38:56.523094 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:38:56.523061 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3859c55_754e_41cf_9ac5_e755a6d94e6d.slice/crio-cab1f5e34149a8ab4535b791d3cb1ba30649dbed033b8e12b0b1badfc84de225 WatchSource:0}: Error finding container cab1f5e34149a8ab4535b791d3cb1ba30649dbed033b8e12b0b1badfc84de225: Status 404 returned error can't find the container with id cab1f5e34149a8ab4535b791d3cb1ba30649dbed033b8e12b0b1badfc84de225 Jul 10 02:38:56.932406 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:56.932376 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-987cdfe4-79q66" event={"ID":"d3859c55-754e-41cf-9ac5-e755a6d94e6d","Type":"ContainerStarted","Data":"868d71f6940affd0e5547c58983453422feffa9c9f24a95601fa279e4c354df9"} Jul 10 02:38:56.932555 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:56.932432 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-987cdfe4-79q66" event={"ID":"d3859c55-754e-41cf-9ac5-e755a6d94e6d","Type":"ContainerStarted","Data":"cab1f5e34149a8ab4535b791d3cb1ba30649dbed033b8e12b0b1badfc84de225"} Jul 10 02:38:56.951743 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:56.951694 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-987cdfe4-79q66" podStartSLOduration=0.951676322 podStartE2EDuration="951.676322ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:38:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:38:56.950886449 +0000 UTC m=+1868.527163560" watchObservedRunningTime="2026-07-10 02:38:56.951676322 +0000 UTC m=+1868.527953407" Jul 10 02:38:57.945190 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:57.945156 2566 generic.go:356] "Generic (PLEG): container finished" podID="d3859c55-754e-41cf-9ac5-e755a6d94e6d" containerID="868d71f6940affd0e5547c58983453422feffa9c9f24a95601fa279e4c354df9" exitCode=7 Jul 10 02:38:57.945582 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:57.945240 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-987cdfe4-79q66" event={"ID":"d3859c55-754e-41cf-9ac5-e755a6d94e6d","Type":"ContainerDied","Data":"868d71f6940affd0e5547c58983453422feffa9c9f24a95601fa279e4c354df9"} Jul 10 02:38:57.945582 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:57.945393 2566 scope.go:117] "RemoveContainer" containerID="868d71f6940affd0e5547c58983453422feffa9c9f24a95601fa279e4c354df9" Jul 10 02:38:58.952662 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:58.952565 2566 generic.go:356] "Generic (PLEG): container finished" podID="d3859c55-754e-41cf-9ac5-e755a6d94e6d" containerID="b0e24031a69d8cc744712b31a4d0e8aefa856f5e8065b821a8436a9bcce35eac" exitCode=7 Jul 10 02:38:58.953009 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:58.952654 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-987cdfe4-79q66" event={"ID":"d3859c55-754e-41cf-9ac5-e755a6d94e6d","Type":"ContainerDied","Data":"b0e24031a69d8cc744712b31a4d0e8aefa856f5e8065b821a8436a9bcce35eac"} Jul 10 02:38:58.953009 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:58.952719 2566 scope.go:117] "RemoveContainer" containerID="868d71f6940affd0e5547c58983453422feffa9c9f24a95601fa279e4c354df9" Jul 10 02:38:58.953009 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:58.952853 2566 scope.go:117] "RemoveContainer" containerID="b0e24031a69d8cc744712b31a4d0e8aefa856f5e8065b821a8436a9bcce35eac" Jul 10 02:38:58.953124 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:38:58.953052 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-987cdfe4-79q66_odh-ai-gateway-infra(d3859c55-754e-41cf-9ac5-e755a6d94e6d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-987cdfe4-79q66" podUID="d3859c55-754e-41cf-9ac5-e755a6d94e6d" Jul 10 02:38:59.960237 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:38:59.960212 2566 scope.go:117] "RemoveContainer" containerID="b0e24031a69d8cc744712b31a4d0e8aefa856f5e8065b821a8436a9bcce35eac" Jul 10 02:38:59.960643 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:38:59.960361 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-987cdfe4-79q66_odh-ai-gateway-infra(d3859c55-754e-41cf-9ac5-e755a6d94e6d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-987cdfe4-79q66" podUID="d3859c55-754e-41cf-9ac5-e755a6d94e6d" Jul 10 02:39:00.931720 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:39:00.931688 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-987cdfe4-67bf7fb44b-vhj2l" Jul 10 02:39:01.663948 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:39:01.663918 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:39:01.764725 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:39:01.764690 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:39:05.301064 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:39:05.301033 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:39:10.518493 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:39:10.518409 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:39:10.618225 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:39:10.618192 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:39:13.277412 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:39:13.277380 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:39:13.377179 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:39:13.377149 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:39:14.005531 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:39:14.005508 2566 scope.go:117] "RemoveContainer" containerID="b0e24031a69d8cc744712b31a4d0e8aefa856f5e8065b821a8436a9bcce35eac" Jul 10 02:39:15.042036 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:39:15.042009 2566 generic.go:356] "Generic (PLEG): container finished" podID="d3859c55-754e-41cf-9ac5-e755a6d94e6d" containerID="ae76133b1e5dd5ff8325e2e74eace1ac6fad990757320ffbf9ba9560aacab2b1" exitCode=0 Jul 10 02:39:15.042457 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:39:15.042090 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-987cdfe4-79q66" event={"ID":"d3859c55-754e-41cf-9ac5-e755a6d94e6d","Type":"ContainerDied","Data":"ae76133b1e5dd5ff8325e2e74eace1ac6fad990757320ffbf9ba9560aacab2b1"} Jul 10 02:39:15.042457 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:39:15.042161 2566 scope.go:117] "RemoveContainer" containerID="b0e24031a69d8cc744712b31a4d0e8aefa856f5e8065b821a8436a9bcce35eac" Jul 10 02:39:16.078708 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:39:16.078674 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-987cdfe4-79q66"] Jul 10 02:39:16.184329 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:39:16.184303 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-987cdfe4-79q66" Jul 10 02:39:16.293336 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:39:16.293302 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3859c55-754e-41cf-9ac5-e755a6d94e6d-maas-api-service-ca\") pod \"d3859c55-754e-41cf-9ac5-e755a6d94e6d\" (UID: \"d3859c55-754e-41cf-9ac5-e755a6d94e6d\") " Jul 10 02:39:16.293746 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:39:16.293720 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3859c55-754e-41cf-9ac5-e755a6d94e6d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d3859c55-754e-41cf-9ac5-e755a6d94e6d" (UID: "d3859c55-754e-41cf-9ac5-e755a6d94e6d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 02:39:16.394042 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:39:16.393956 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3859c55-754e-41cf-9ac5-e755a6d94e6d-maas-api-service-ca\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:39:17.058881 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:39:17.058855 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-987cdfe4-79q66" Jul 10 02:39:17.059099 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:39:17.058883 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-987cdfe4-79q66" event={"ID":"d3859c55-754e-41cf-9ac5-e755a6d94e6d","Type":"ContainerDied","Data":"cab1f5e34149a8ab4535b791d3cb1ba30649dbed033b8e12b0b1badfc84de225"} Jul 10 02:39:17.059099 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:39:17.058936 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cab1f5e34149a8ab4535b791d3cb1ba30649dbed033b8e12b0b1badfc84de225" Jul 10 02:39:17.099190 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:39:17.099160 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-987cdfe4-79q66"] Jul 10 02:39:17.105572 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:39:17.105549 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-987cdfe4-79q66"] Jul 10 02:39:19.011571 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:39:19.011531 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3859c55-754e-41cf-9ac5-e755a6d94e6d" path="/var/lib/kubelet/pods/d3859c55-754e-41cf-9ac5-e755a6d94e6d/volumes" Jul 10 02:39:56.128662 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:39:56.128631 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-987cdfe4-75lj2"] Jul 10 02:39:56.131012 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:39:56.128973 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3859c55-754e-41cf-9ac5-e755a6d94e6d" containerName="revoke-keys" Jul 10 02:39:56.131012 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:39:56.128984 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3859c55-754e-41cf-9ac5-e755a6d94e6d" containerName="revoke-keys" Jul 10 02:39:56.131012 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:39:56.128998 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3859c55-754e-41cf-9ac5-e755a6d94e6d" containerName="revoke-keys" Jul 10 02:39:56.131012 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:39:56.129004 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3859c55-754e-41cf-9ac5-e755a6d94e6d" containerName="revoke-keys" Jul 10 02:39:56.131012 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:39:56.129012 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3859c55-754e-41cf-9ac5-e755a6d94e6d" containerName="revoke-keys" Jul 10 02:39:56.131012 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:39:56.129017 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3859c55-754e-41cf-9ac5-e755a6d94e6d" containerName="revoke-keys" Jul 10 02:39:56.131012 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:39:56.129076 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3859c55-754e-41cf-9ac5-e755a6d94e6d" containerName="revoke-keys" Jul 10 02:39:56.131012 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:39:56.129084 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3859c55-754e-41cf-9ac5-e755a6d94e6d" containerName="revoke-keys" Jul 10 02:39:56.131905 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:39:56.131889 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-987cdfe4-75lj2" Jul 10 02:39:56.134530 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:39:56.134509 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rppvw\"" Jul 10 02:39:56.140373 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:39:56.140347 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-987cdfe4-75lj2"] Jul 10 02:39:56.218777 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:39:56.218746 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:39:56.240139 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:39:56.240113 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b99df84c-c677-4cb5-8b24-1149b5092624-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-987cdfe4-75lj2\" (UID: \"b99df84c-c677-4cb5-8b24-1149b5092624\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-987cdfe4-75lj2" Jul 10 02:39:56.321179 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:39:56.321153 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:39:56.341085 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:39:56.341059 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b99df84c-c677-4cb5-8b24-1149b5092624-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-987cdfe4-75lj2\" (UID: \"b99df84c-c677-4cb5-8b24-1149b5092624\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-987cdfe4-75lj2" Jul 10 02:39:56.341681 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:39:56.341663 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b99df84c-c677-4cb5-8b24-1149b5092624-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-987cdfe4-75lj2\" (UID: \"b99df84c-c677-4cb5-8b24-1149b5092624\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-987cdfe4-75lj2" Jul 10 02:39:56.442507 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:39:56.442437 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-987cdfe4-75lj2" Jul 10 02:39:56.779537 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:39:56.779508 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-987cdfe4-75lj2"] Jul 10 02:39:56.781099 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:39:56.781071 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb99df84c_c677_4cb5_8b24_1149b5092624.slice/crio-76defaf415df602680e1c155d7f2469f07d9fd184a67cddec9797098019f3b62 WatchSource:0}: Error finding container 76defaf415df602680e1c155d7f2469f07d9fd184a67cddec9797098019f3b62: Status 404 returned error can't find the container with id 76defaf415df602680e1c155d7f2469f07d9fd184a67cddec9797098019f3b62 Jul 10 02:39:57.285496 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:39:57.285463 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-987cdfe4-75lj2" event={"ID":"b99df84c-c677-4cb5-8b24-1149b5092624","Type":"ContainerStarted","Data":"04cef262f555a9b90ea7f8158cd11f57e8a0d08a2b695b23714379b39aa31803"} Jul 10 02:39:57.285961 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:39:57.285512 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-987cdfe4-75lj2" event={"ID":"b99df84c-c677-4cb5-8b24-1149b5092624","Type":"ContainerStarted","Data":"76defaf415df602680e1c155d7f2469f07d9fd184a67cddec9797098019f3b62"} Jul 10 02:39:57.303230 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:39:57.303184 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-987cdfe4-75lj2" podStartSLOduration=1.30316848 podStartE2EDuration="1.30316848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:39:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:39:57.301206301 +0000 UTC m=+1928.877483410" watchObservedRunningTime="2026-07-10 02:39:57.30316848 +0000 UTC m=+1928.879445601" Jul 10 02:39:58.292414 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:39:58.292379 2566 generic.go:356] "Generic (PLEG): container finished" podID="b99df84c-c677-4cb5-8b24-1149b5092624" containerID="04cef262f555a9b90ea7f8158cd11f57e8a0d08a2b695b23714379b39aa31803" exitCode=0 Jul 10 02:39:58.292829 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:39:58.292473 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-987cdfe4-75lj2" event={"ID":"b99df84c-c677-4cb5-8b24-1149b5092624","Type":"ContainerDied","Data":"04cef262f555a9b90ea7f8158cd11f57e8a0d08a2b695b23714379b39aa31803"} Jul 10 02:39:59.442368 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:39:59.442339 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-987cdfe4-75lj2" Jul 10 02:39:59.568210 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:39:59.568113 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b99df84c-c677-4cb5-8b24-1149b5092624-maas-api-service-ca\") pod \"b99df84c-c677-4cb5-8b24-1149b5092624\" (UID: \"b99df84c-c677-4cb5-8b24-1149b5092624\") " Jul 10 02:39:59.568548 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:39:59.568516 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b99df84c-c677-4cb5-8b24-1149b5092624-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b99df84c-c677-4cb5-8b24-1149b5092624" (UID: "b99df84c-c677-4cb5-8b24-1149b5092624"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 02:39:59.669208 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:39:59.669155 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b99df84c-c677-4cb5-8b24-1149b5092624-maas-api-service-ca\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:40:00.304794 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:40:00.304767 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-987cdfe4-75lj2" Jul 10 02:40:00.304980 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:40:00.304806 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-987cdfe4-75lj2" event={"ID":"b99df84c-c677-4cb5-8b24-1149b5092624","Type":"ContainerDied","Data":"76defaf415df602680e1c155d7f2469f07d9fd184a67cddec9797098019f3b62"} Jul 10 02:40:00.304980 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:40:00.304853 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76defaf415df602680e1c155d7f2469f07d9fd184a67cddec9797098019f3b62" Jul 10 02:40:01.486355 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:40:01.486325 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-987cdfe4-67bf7fb44b-vhj2l"] Jul 10 02:40:01.486852 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:40:01.486566 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-987cdfe4-67bf7fb44b-vhj2l" podUID="eed197ab-a025-42dc-a504-05ecb9612957" containerName="maas-api" containerID="cri-o://393ddef7d5e8777cdbee94b556cb6f3cd4d7fec12e8b4f6ffdc3784ec633a5dd" gracePeriod=30 Jul 10 02:40:01.511946 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:40:01.511904 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:40:01.732492 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:40:01.732469 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-987cdfe4-67bf7fb44b-vhj2l" Jul 10 02:40:01.788461 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:40:01.788438 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eed197ab-a025-42dc-a504-05ecb9612957-maas-api-tls\") pod \"eed197ab-a025-42dc-a504-05ecb9612957\" (UID: \"eed197ab-a025-42dc-a504-05ecb9612957\") " Jul 10 02:40:01.788584 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:40:01.788504 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rfh9\" (UniqueName: \"kubernetes.io/projected/eed197ab-a025-42dc-a504-05ecb9612957-kube-api-access-5rfh9\") pod \"eed197ab-a025-42dc-a504-05ecb9612957\" (UID: \"eed197ab-a025-42dc-a504-05ecb9612957\") " Jul 10 02:40:01.788584 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:40:01.788563 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eed197ab-a025-42dc-a504-05ecb9612957-openshift-service-ca\") pod \"eed197ab-a025-42dc-a504-05ecb9612957\" (UID: \"eed197ab-a025-42dc-a504-05ecb9612957\") " Jul 10 02:40:01.788934 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:40:01.788903 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eed197ab-a025-42dc-a504-05ecb9612957-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "eed197ab-a025-42dc-a504-05ecb9612957" (UID: "eed197ab-a025-42dc-a504-05ecb9612957"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 02:40:01.790640 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:40:01.790618 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eed197ab-a025-42dc-a504-05ecb9612957-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "eed197ab-a025-42dc-a504-05ecb9612957" (UID: "eed197ab-a025-42dc-a504-05ecb9612957"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 02:40:01.790728 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:40:01.790635 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eed197ab-a025-42dc-a504-05ecb9612957-kube-api-access-5rfh9" (OuterVolumeSpecName: "kube-api-access-5rfh9") pod "eed197ab-a025-42dc-a504-05ecb9612957" (UID: "eed197ab-a025-42dc-a504-05ecb9612957"). InnerVolumeSpecName "kube-api-access-5rfh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 02:40:01.889727 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:40:01.889700 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eed197ab-a025-42dc-a504-05ecb9612957-maas-api-tls\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:40:01.889727 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:40:01.889723 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rfh9\" (UniqueName: \"kubernetes.io/projected/eed197ab-a025-42dc-a504-05ecb9612957-kube-api-access-5rfh9\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:40:01.889857 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:40:01.889738 2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eed197ab-a025-42dc-a504-05ecb9612957-openshift-service-ca\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:40:02.316288 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:40:02.316251 2566 generic.go:356] "Generic (PLEG): container finished" podID="eed197ab-a025-42dc-a504-05ecb9612957" containerID="393ddef7d5e8777cdbee94b556cb6f3cd4d7fec12e8b4f6ffdc3784ec633a5dd" exitCode=0 Jul 10 02:40:02.316456 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:40:02.316297 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-987cdfe4-67bf7fb44b-vhj2l" Jul 10 02:40:02.316456 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:40:02.316338 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-987cdfe4-67bf7fb44b-vhj2l" event={"ID":"eed197ab-a025-42dc-a504-05ecb9612957","Type":"ContainerDied","Data":"393ddef7d5e8777cdbee94b556cb6f3cd4d7fec12e8b4f6ffdc3784ec633a5dd"} Jul 10 02:40:02.316456 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:40:02.316390 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-987cdfe4-67bf7fb44b-vhj2l" event={"ID":"eed197ab-a025-42dc-a504-05ecb9612957","Type":"ContainerDied","Data":"de9fec17864702b8a51a34ffae9df01bf1c83ac99deed85de80231cc71897516"} Jul 10 02:40:02.316456 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:40:02.316414 2566 scope.go:117] "RemoveContainer" containerID="393ddef7d5e8777cdbee94b556cb6f3cd4d7fec12e8b4f6ffdc3784ec633a5dd" Jul 10 02:40:02.326399 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:40:02.326378 2566 scope.go:117] "RemoveContainer" containerID="393ddef7d5e8777cdbee94b556cb6f3cd4d7fec12e8b4f6ffdc3784ec633a5dd" Jul 10 02:40:02.326647 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:40:02.326627 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"393ddef7d5e8777cdbee94b556cb6f3cd4d7fec12e8b4f6ffdc3784ec633a5dd\": container with ID starting with 393ddef7d5e8777cdbee94b556cb6f3cd4d7fec12e8b4f6ffdc3784ec633a5dd not found: ID does not exist" containerID="393ddef7d5e8777cdbee94b556cb6f3cd4d7fec12e8b4f6ffdc3784ec633a5dd" Jul 10 02:40:02.326705 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:40:02.326656 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"393ddef7d5e8777cdbee94b556cb6f3cd4d7fec12e8b4f6ffdc3784ec633a5dd"} err="failed to get container status \"393ddef7d5e8777cdbee94b556cb6f3cd4d7fec12e8b4f6ffdc3784ec633a5dd\": rpc error: code = NotFound desc = could not find container \"393ddef7d5e8777cdbee94b556cb6f3cd4d7fec12e8b4f6ffdc3784ec633a5dd\": container with ID starting with 393ddef7d5e8777cdbee94b556cb6f3cd4d7fec12e8b4f6ffdc3784ec633a5dd not found: ID does not exist" Jul 10 02:40:02.342398 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:40:02.342374 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-987cdfe4-67bf7fb44b-vhj2l"] Jul 10 02:40:02.348220 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:40:02.348198 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-987cdfe4-67bf7fb44b-vhj2l"] Jul 10 02:40:02.690929 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:40:02.690842 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:40:02.889121 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:40:02.889091 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:40:03.011458 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:40:03.011379 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eed197ab-a025-42dc-a504-05ecb9612957" path="/var/lib/kubelet/pods/eed197ab-a025-42dc-a504-05ecb9612957/volumes" Jul 10 02:40:08.755717 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:40:08.755684 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:40:08.855613 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:40:08.855579 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:40:09.278959 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:40:09.278853 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:40:09.379571 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:40:09.379538 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:40:09.482689 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:40:09.482659 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:40:11.022440 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:40:11.022404 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:40:17.442636 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:40:17.442585 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:40:17.503795 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:40:17.503762 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:40:17.593550 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:40:17.593521 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:40:17.698822 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:40:17.698746 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:40:17.795746 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:40:17.795711 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:40:18.198526 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:40:18.198494 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:40:44.234859 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:40:44.234775 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:40:50.192047 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:40:50.192015 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:40:50.294551 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:40:50.294522 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:40:50.396174 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:40:50.396140 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:41:06.356156 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:06.356120 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:41:12.273114 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:12.273081 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:41:12.375221 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:12.375189 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:41:12.473866 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:12.473830 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:41:18.480745 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:18.480709 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:41:18.536733 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:18.536696 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:41:23.047350 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:23.047318 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:41:23.871135 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:23.871098 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:41:30.011842 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:30.011806 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:41:30.119555 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:30.119507 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:41:30.216173 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:30.216133 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:41:30.312947 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:30.312913 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:41:31.127221 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:31.127165 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:41:31.227832 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:31.227798 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:41:31.327138 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:31.327104 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:41:33.011375 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:33.011345 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:41:39.077277 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:39.077236 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:41:39.178651 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:39.178588 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:41:39.781530 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:39.781487 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:41:45.362318 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:45.362280 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:41:45.414528 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:45.414495 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:41:46.085384 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:46.085352 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:41:46.237829 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:46.237803 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:41:46.333588 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:46.333557 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:41:46.670958 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:46.670921 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:41:47.297566 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:47.297535 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:41:49.520339 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:49.520308 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:41:49.622610 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:49.622562 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:41:49.666526 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:49.666495 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-abb8864d-549b96c8fc-xz7gp"] Jul 10 02:41:49.666866 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:49.666854 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eed197ab-a025-42dc-a504-05ecb9612957" containerName="maas-api" Jul 10 02:41:49.666919 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:49.666868 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="eed197ab-a025-42dc-a504-05ecb9612957" containerName="maas-api" Jul 10 02:41:49.666919 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:49.666881 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b99df84c-c677-4cb5-8b24-1149b5092624" containerName="revoke-keys" Jul 10 02:41:49.666919 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:49.666886 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b99df84c-c677-4cb5-8b24-1149b5092624" containerName="revoke-keys" Jul 10 02:41:49.667012 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:49.666949 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b99df84c-c677-4cb5-8b24-1149b5092624" containerName="revoke-keys" Jul 10 02:41:49.667012 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:49.666956 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="eed197ab-a025-42dc-a504-05ecb9612957" containerName="maas-api" Jul 10 02:41:49.667012 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:49.666964 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3859c55-754e-41cf-9ac5-e755a6d94e6d" containerName="revoke-keys" Jul 10 02:41:49.670100 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:49.670085 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-abb8864d-549b96c8fc-xz7gp" Jul 10 02:41:49.672938 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:49.672913 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-abb8864d-serving-cert\"" Jul 10 02:41:49.672938 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:49.672928 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-r7s9m\"" Jul 10 02:41:49.673117 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:49.672969 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 02:41:49.673117 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:49.673013 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 10 02:41:49.680384 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:49.680359 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-abb8864d-549b96c8fc-xz7gp"] Jul 10 02:41:49.736501 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:49.736468 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snxkq\" (UniqueName: \"kubernetes.io/projected/89273dd2-1f06-4737-8a2d-d2ed48cabdcc-kube-api-access-snxkq\") pod \"maas-api-e2e-mt-abb8864d-549b96c8fc-xz7gp\" (UID: \"89273dd2-1f06-4737-8a2d-d2ed48cabdcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-abb8864d-549b96c8fc-xz7gp" Jul 10 02:41:49.736688 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:49.736525 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89273dd2-1f06-4737-8a2d-d2ed48cabdcc-maas-api-tls\") pod \"maas-api-e2e-mt-abb8864d-549b96c8fc-xz7gp\" (UID: \"89273dd2-1f06-4737-8a2d-d2ed48cabdcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-abb8864d-549b96c8fc-xz7gp" Jul 10 02:41:49.736688 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:49.736644 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89273dd2-1f06-4737-8a2d-d2ed48cabdcc-openshift-service-ca\") pod \"maas-api-e2e-mt-abb8864d-549b96c8fc-xz7gp\" (UID: \"89273dd2-1f06-4737-8a2d-d2ed48cabdcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-abb8864d-549b96c8fc-xz7gp" Jul 10 02:41:49.757895 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:49.757872 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:41:49.838200 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:49.838107 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snxkq\" (UniqueName: \"kubernetes.io/projected/89273dd2-1f06-4737-8a2d-d2ed48cabdcc-kube-api-access-snxkq\") pod \"maas-api-e2e-mt-abb8864d-549b96c8fc-xz7gp\" (UID: \"89273dd2-1f06-4737-8a2d-d2ed48cabdcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-abb8864d-549b96c8fc-xz7gp" Jul 10 02:41:49.838200 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:49.838188 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89273dd2-1f06-4737-8a2d-d2ed48cabdcc-maas-api-tls\") pod \"maas-api-e2e-mt-abb8864d-549b96c8fc-xz7gp\" (UID: \"89273dd2-1f06-4737-8a2d-d2ed48cabdcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-abb8864d-549b96c8fc-xz7gp" Jul 10 02:41:49.838454 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:49.838222 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89273dd2-1f06-4737-8a2d-d2ed48cabdcc-openshift-service-ca\") pod \"maas-api-e2e-mt-abb8864d-549b96c8fc-xz7gp\" (UID: \"89273dd2-1f06-4737-8a2d-d2ed48cabdcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-abb8864d-549b96c8fc-xz7gp" Jul 10 02:41:49.838454 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:41:49.838325 2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-abb8864d-serving-cert: secret "maas-api-e2e-mt-abb8864d-serving-cert" not found Jul 10 02:41:49.838454 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:41:49.838402 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89273dd2-1f06-4737-8a2d-d2ed48cabdcc-maas-api-tls podName:89273dd2-1f06-4737-8a2d-d2ed48cabdcc nodeName:}" failed. No retries permitted until 2026-07-10 02:41:50.338380578 +0000 UTC m=+2041.914657646 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/89273dd2-1f06-4737-8a2d-d2ed48cabdcc-maas-api-tls") pod "maas-api-e2e-mt-abb8864d-549b96c8fc-xz7gp" (UID: "89273dd2-1f06-4737-8a2d-d2ed48cabdcc") : secret "maas-api-e2e-mt-abb8864d-serving-cert" not found Jul 10 02:41:49.838852 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:49.838836 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89273dd2-1f06-4737-8a2d-d2ed48cabdcc-openshift-service-ca\") pod \"maas-api-e2e-mt-abb8864d-549b96c8fc-xz7gp\" (UID: \"89273dd2-1f06-4737-8a2d-d2ed48cabdcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-abb8864d-549b96c8fc-xz7gp" Jul 10 02:41:49.850643 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:49.850616 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snxkq\" (UniqueName: \"kubernetes.io/projected/89273dd2-1f06-4737-8a2d-d2ed48cabdcc-kube-api-access-snxkq\") pod \"maas-api-e2e-mt-abb8864d-549b96c8fc-xz7gp\" (UID: \"89273dd2-1f06-4737-8a2d-d2ed48cabdcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-abb8864d-549b96c8fc-xz7gp" Jul 10 02:41:50.115430 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:50.115353 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:41:50.331537 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:50.331508 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:41:50.342761 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:50.342735 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89273dd2-1f06-4737-8a2d-d2ed48cabdcc-maas-api-tls\") pod \"maas-api-e2e-mt-abb8864d-549b96c8fc-xz7gp\" (UID: \"89273dd2-1f06-4737-8a2d-d2ed48cabdcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-abb8864d-549b96c8fc-xz7gp" Jul 10 02:41:50.345352 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:50.345330 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89273dd2-1f06-4737-8a2d-d2ed48cabdcc-maas-api-tls\") pod \"maas-api-e2e-mt-abb8864d-549b96c8fc-xz7gp\" (UID: \"89273dd2-1f06-4737-8a2d-d2ed48cabdcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-abb8864d-549b96c8fc-xz7gp" Jul 10 02:41:50.432877 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:50.432780 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:41:50.581761 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:50.581716 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-abb8864d-549b96c8fc-xz7gp" Jul 10 02:41:50.719652 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:50.719622 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-abb8864d-549b96c8fc-xz7gp"] Jul 10 02:41:50.721640 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:41:50.721608 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89273dd2_1f06_4737_8a2d_d2ed48cabdcc.slice/crio-3b92fb126eb389bf3b158b845a0d7870cf7f52581fa5adb1546d396d06116a8e WatchSource:0}: Error finding container 3b92fb126eb389bf3b158b845a0d7870cf7f52581fa5adb1546d396d06116a8e: Status 404 returned error can't find the container with id 3b92fb126eb389bf3b158b845a0d7870cf7f52581fa5adb1546d396d06116a8e Jul 10 02:41:50.905973 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:50.905937 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-abb8864d-549b96c8fc-xz7gp" event={"ID":"89273dd2-1f06-4737-8a2d-d2ed48cabdcc","Type":"ContainerStarted","Data":"3b92fb126eb389bf3b158b845a0d7870cf7f52581fa5adb1546d396d06116a8e"} Jul 10 02:41:51.913041 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:51.913007 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-abb8864d-549b96c8fc-xz7gp" event={"ID":"89273dd2-1f06-4737-8a2d-d2ed48cabdcc","Type":"ContainerStarted","Data":"5e481b9bfe09a9f5291a0578e2e2f3e525a07c045295b5132446b2a2665b1c94"} Jul 10 02:41:51.913453 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:51.913132 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-abb8864d-549b96c8fc-xz7gp" Jul 10 02:41:51.932952 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:51.932891 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-abb8864d-549b96c8fc-xz7gp" podStartSLOduration=2.419449359 podStartE2EDuration="2.932874652s" totalImagesPullingTime="513.425293ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:41:51.931909913 +0000 UTC m=+2043.508187002" watchObservedRunningTime="2026-07-10 02:41:51.932874652 +0000 UTC m=+2043.509151738" Jul 10 02:41:54.050355 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:54.050315 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:41:54.733529 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:54.733495 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:41:54.832979 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:54.832946 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:41:55.200797 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:55.200761 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:41:57.924048 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:41:57.924016 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-abb8864d-549b96c8fc-xz7gp" Jul 10 02:42:01.522852 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:01.522817 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:42:01.583575 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:01.583544 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:42:01.907355 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:01.907273 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-abb8864d-ssmpn"] Jul 10 02:42:01.910937 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:01.910920 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-abb8864d-ssmpn" Jul 10 02:42:01.913530 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:01.913509 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rppvw\"" Jul 10 02:42:01.918800 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:01.918723 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-abb8864d-ssmpn"] Jul 10 02:42:02.048440 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:02.048401 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2a7ed0-25ee-4f2b-9fce-c1d98e0205f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-abb8864d-ssmpn\" (UID: \"ec2a7ed0-25ee-4f2b-9fce-c1d98e0205f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-abb8864d-ssmpn" Jul 10 02:42:02.149001 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:02.148969 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2a7ed0-25ee-4f2b-9fce-c1d98e0205f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-abb8864d-ssmpn\" (UID: \"ec2a7ed0-25ee-4f2b-9fce-c1d98e0205f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-abb8864d-ssmpn" Jul 10 02:42:02.149650 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:02.149624 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2a7ed0-25ee-4f2b-9fce-c1d98e0205f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-abb8864d-ssmpn\" (UID: \"ec2a7ed0-25ee-4f2b-9fce-c1d98e0205f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-abb8864d-ssmpn" Jul 10 02:42:02.222636 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:02.222530 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-abb8864d-ssmpn" Jul 10 02:42:02.355787 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:02.355764 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-abb8864d-ssmpn"] Jul 10 02:42:02.358178 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:42:02.358122 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec2a7ed0_25ee_4f2b_9fce_c1d98e0205f0.slice/crio-e3056d3b34b13a9795964ff2225f18afe6608e8688a4c4a690041a22fe404301 WatchSource:0}: Error finding container e3056d3b34b13a9795964ff2225f18afe6608e8688a4c4a690041a22fe404301: Status 404 returned error can't find the container with id e3056d3b34b13a9795964ff2225f18afe6608e8688a4c4a690041a22fe404301 Jul 10 02:42:02.976713 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:02.976626 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-abb8864d-ssmpn" event={"ID":"ec2a7ed0-25ee-4f2b-9fce-c1d98e0205f0","Type":"ContainerStarted","Data":"74e29f3247de64333f301c253ec00f9eac5f9afddec6c67592fe01861a42f577"} Jul 10 02:42:02.976713 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:02.976683 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-abb8864d-ssmpn" event={"ID":"ec2a7ed0-25ee-4f2b-9fce-c1d98e0205f0","Type":"ContainerStarted","Data":"e3056d3b34b13a9795964ff2225f18afe6608e8688a4c4a690041a22fe404301"} Jul 10 02:42:02.996758 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:02.996705 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-abb8864d-ssmpn" podStartSLOduration=1.99669132 podStartE2EDuration="1.99669132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:42:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:42:02.993084853 +0000 UTC m=+2054.569361964" watchObservedRunningTime="2026-07-10 02:42:02.99669132 +0000 UTC m=+2054.572968405" Jul 10 02:42:03.984583 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:03.984542 2566 generic.go:356] "Generic (PLEG): container finished" podID="ec2a7ed0-25ee-4f2b-9fce-c1d98e0205f0" containerID="74e29f3247de64333f301c253ec00f9eac5f9afddec6c67592fe01861a42f577" exitCode=0 Jul 10 02:42:03.984996 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:03.984635 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-abb8864d-ssmpn" event={"ID":"ec2a7ed0-25ee-4f2b-9fce-c1d98e0205f0","Type":"ContainerDied","Data":"74e29f3247de64333f301c253ec00f9eac5f9afddec6c67592fe01861a42f577"} Jul 10 02:42:05.125455 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:05.125429 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-abb8864d-ssmpn" Jul 10 02:42:05.277276 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:05.277245 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2a7ed0-25ee-4f2b-9fce-c1d98e0205f0-maas-api-service-ca\") pod \"ec2a7ed0-25ee-4f2b-9fce-c1d98e0205f0\" (UID: \"ec2a7ed0-25ee-4f2b-9fce-c1d98e0205f0\") " Jul 10 02:42:05.277665 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:05.277643 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec2a7ed0-25ee-4f2b-9fce-c1d98e0205f0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ec2a7ed0-25ee-4f2b-9fce-c1d98e0205f0" (UID: "ec2a7ed0-25ee-4f2b-9fce-c1d98e0205f0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 02:42:05.378454 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:05.378419 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2a7ed0-25ee-4f2b-9fce-c1d98e0205f0-maas-api-service-ca\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:42:05.998362 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:05.998333 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-abb8864d-ssmpn" Jul 10 02:42:05.998564 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:05.998377 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-abb8864d-ssmpn" event={"ID":"ec2a7ed0-25ee-4f2b-9fce-c1d98e0205f0","Type":"ContainerDied","Data":"e3056d3b34b13a9795964ff2225f18afe6608e8688a4c4a690041a22fe404301"} Jul 10 02:42:05.998564 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:05.998428 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3056d3b34b13a9795964ff2225f18afe6608e8688a4c4a690041a22fe404301" Jul 10 02:42:12.411460 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:12.411381 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-abb8864d-549b96c8fc-xz7gp"] Jul 10 02:42:12.411902 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:12.411584 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-abb8864d-549b96c8fc-xz7gp" podUID="89273dd2-1f06-4737-8a2d-d2ed48cabdcc" containerName="maas-api" containerID="cri-o://5e481b9bfe09a9f5291a0578e2e2f3e525a07c045295b5132446b2a2665b1c94" gracePeriod=30 Jul 10 02:42:12.440211 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:12.440173 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:42:12.664107 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:12.664051 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-abb8864d-549b96c8fc-xz7gp" Jul 10 02:42:12.746328 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:12.746301 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89273dd2-1f06-4737-8a2d-d2ed48cabdcc-maas-api-tls\") pod \"89273dd2-1f06-4737-8a2d-d2ed48cabdcc\" (UID: \"89273dd2-1f06-4737-8a2d-d2ed48cabdcc\") " Jul 10 02:42:12.746489 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:12.746358 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89273dd2-1f06-4737-8a2d-d2ed48cabdcc-openshift-service-ca\") pod \"89273dd2-1f06-4737-8a2d-d2ed48cabdcc\" (UID: \"89273dd2-1f06-4737-8a2d-d2ed48cabdcc\") " Jul 10 02:42:12.746489 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:12.746432 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-snxkq\" (UniqueName: \"kubernetes.io/projected/89273dd2-1f06-4737-8a2d-d2ed48cabdcc-kube-api-access-snxkq\") pod \"89273dd2-1f06-4737-8a2d-d2ed48cabdcc\" (UID: \"89273dd2-1f06-4737-8a2d-d2ed48cabdcc\") " Jul 10 02:42:12.746755 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:12.746735 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89273dd2-1f06-4737-8a2d-d2ed48cabdcc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "89273dd2-1f06-4737-8a2d-d2ed48cabdcc" (UID: "89273dd2-1f06-4737-8a2d-d2ed48cabdcc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 02:42:12.748504 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:12.748477 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89273dd2-1f06-4737-8a2d-d2ed48cabdcc-kube-api-access-snxkq" (OuterVolumeSpecName: "kube-api-access-snxkq") pod "89273dd2-1f06-4737-8a2d-d2ed48cabdcc" (UID: "89273dd2-1f06-4737-8a2d-d2ed48cabdcc"). InnerVolumeSpecName "kube-api-access-snxkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 02:42:12.748640 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:12.748520 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89273dd2-1f06-4737-8a2d-d2ed48cabdcc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "89273dd2-1f06-4737-8a2d-d2ed48cabdcc" (UID: "89273dd2-1f06-4737-8a2d-d2ed48cabdcc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 02:42:12.847585 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:12.847554 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snxkq\" (UniqueName: \"kubernetes.io/projected/89273dd2-1f06-4737-8a2d-d2ed48cabdcc-kube-api-access-snxkq\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:42:12.847585 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:12.847580 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89273dd2-1f06-4737-8a2d-d2ed48cabdcc-maas-api-tls\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:42:12.847585 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:12.847589 2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89273dd2-1f06-4737-8a2d-d2ed48cabdcc-openshift-service-ca\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:42:13.039935 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:13.039901 2566 generic.go:356] "Generic (PLEG): container finished" podID="89273dd2-1f06-4737-8a2d-d2ed48cabdcc" containerID="5e481b9bfe09a9f5291a0578e2e2f3e525a07c045295b5132446b2a2665b1c94" exitCode=0 Jul 10 02:42:13.040082 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:13.039943 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-abb8864d-549b96c8fc-xz7gp" Jul 10 02:42:13.040082 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:13.039980 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-abb8864d-549b96c8fc-xz7gp" event={"ID":"89273dd2-1f06-4737-8a2d-d2ed48cabdcc","Type":"ContainerDied","Data":"5e481b9bfe09a9f5291a0578e2e2f3e525a07c045295b5132446b2a2665b1c94"} Jul 10 02:42:13.040082 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:13.040029 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-abb8864d-549b96c8fc-xz7gp" event={"ID":"89273dd2-1f06-4737-8a2d-d2ed48cabdcc","Type":"ContainerDied","Data":"3b92fb126eb389bf3b158b845a0d7870cf7f52581fa5adb1546d396d06116a8e"} Jul 10 02:42:13.040082 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:13.040052 2566 scope.go:117] "RemoveContainer" containerID="5e481b9bfe09a9f5291a0578e2e2f3e525a07c045295b5132446b2a2665b1c94" Jul 10 02:42:13.049263 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:13.049241 2566 scope.go:117] "RemoveContainer" containerID="5e481b9bfe09a9f5291a0578e2e2f3e525a07c045295b5132446b2a2665b1c94" Jul 10 02:42:13.049546 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:42:13.049525 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e481b9bfe09a9f5291a0578e2e2f3e525a07c045295b5132446b2a2665b1c94\": container with ID starting with 5e481b9bfe09a9f5291a0578e2e2f3e525a07c045295b5132446b2a2665b1c94 not found: ID does not exist" containerID="5e481b9bfe09a9f5291a0578e2e2f3e525a07c045295b5132446b2a2665b1c94" Jul 10 02:42:13.049635 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:13.049553 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e481b9bfe09a9f5291a0578e2e2f3e525a07c045295b5132446b2a2665b1c94"} err="failed to get container status \"5e481b9bfe09a9f5291a0578e2e2f3e525a07c045295b5132446b2a2665b1c94\": rpc error: code = NotFound desc = could not find container \"5e481b9bfe09a9f5291a0578e2e2f3e525a07c045295b5132446b2a2665b1c94\": container with ID starting with 5e481b9bfe09a9f5291a0578e2e2f3e525a07c045295b5132446b2a2665b1c94 not found: ID does not exist" Jul 10 02:42:13.067452 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:13.067416 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-abb8864d-549b96c8fc-xz7gp"] Jul 10 02:42:13.069953 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:13.069929 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-abb8864d-549b96c8fc-xz7gp"] Jul 10 02:42:15.010805 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:15.010763 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89273dd2-1f06-4737-8a2d-d2ed48cabdcc" path="/var/lib/kubelet/pods/89273dd2-1f06-4737-8a2d-d2ed48cabdcc/volumes" Jul 10 02:42:17.004635 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:17.004583 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:42:23.545970 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:23.545937 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:42:23.608373 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:23.608335 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:42:27.730831 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:27.730796 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:42:27.982330 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:27.982231 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:42:28.113397 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:28.113364 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:42:28.213254 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:28.213225 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:42:28.361570 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:28.361531 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:42:34.393836 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:34.393805 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:42:34.455767 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:34.455734 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:42:34.532062 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:34.532018 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:42:34.633071 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:34.633038 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:42:34.771318 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:34.771286 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:42:41.121627 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:41.121559 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:42:41.188491 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:41.188447 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:42:48.139659 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:48.139622 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:42:48.497378 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:48.497303 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:42:49.056023 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:49.055995 2566 scope.go:117] "RemoveContainer" containerID="65529094217b56135a1948a0b3a3bf2ae660c662b2a9c21dc23cc4397e76d1f2" Jul 10 02:42:49.184426 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:49.184400 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:42:51.704871 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:51.704839 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:42:51.784679 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:51.784640 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:42:51.785105 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:51.785080 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b3d83c99-7f5cccd68f-vmflp"] Jul 10 02:42:51.785656 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:51.785637 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89273dd2-1f06-4737-8a2d-d2ed48cabdcc" containerName="maas-api" Jul 10 02:42:51.785733 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:51.785659 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="89273dd2-1f06-4737-8a2d-d2ed48cabdcc" containerName="maas-api" Jul 10 02:42:51.785733 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:51.785689 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec2a7ed0-25ee-4f2b-9fce-c1d98e0205f0" containerName="revoke-keys" Jul 10 02:42:51.785733 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:51.785697 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec2a7ed0-25ee-4f2b-9fce-c1d98e0205f0" containerName="revoke-keys" Jul 10 02:42:51.785887 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:51.785806 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec2a7ed0-25ee-4f2b-9fce-c1d98e0205f0" containerName="revoke-keys" Jul 10 02:42:51.785887 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:51.785820 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="89273dd2-1f06-4737-8a2d-d2ed48cabdcc" containerName="maas-api" Jul 10 02:42:51.790862 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:51.790843 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b3d83c99-7f5cccd68f-vmflp" Jul 10 02:42:51.795571 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:51.795549 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 02:42:51.796371 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:51.796354 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-b3d83c99-serving-cert\"" Jul 10 02:42:51.796708 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:51.796686 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 10 02:42:51.800319 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:51.800289 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-r7s9m\"" Jul 10 02:42:51.803386 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:51.803367 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b3d83c99-7f5cccd68f-vmflp"] Jul 10 02:42:51.894086 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:51.894057 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:42:51.983925 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:51.983851 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3110905d-14cd-46a3-9ee0-cda2a2da63aa-maas-api-tls\") pod \"maas-api-e2e-mt-b3d83c99-7f5cccd68f-vmflp\" (UID: \"3110905d-14cd-46a3-9ee0-cda2a2da63aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b3d83c99-7f5cccd68f-vmflp" Jul 10 02:42:51.983925 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:51.983896 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3110905d-14cd-46a3-9ee0-cda2a2da63aa-openshift-service-ca\") pod \"maas-api-e2e-mt-b3d83c99-7f5cccd68f-vmflp\" (UID: \"3110905d-14cd-46a3-9ee0-cda2a2da63aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b3d83c99-7f5cccd68f-vmflp" Jul 10 02:42:51.983925 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:51.983920 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxqqk\" (UniqueName: \"kubernetes.io/projected/3110905d-14cd-46a3-9ee0-cda2a2da63aa-kube-api-access-hxqqk\") pod \"maas-api-e2e-mt-b3d83c99-7f5cccd68f-vmflp\" (UID: \"3110905d-14cd-46a3-9ee0-cda2a2da63aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b3d83c99-7f5cccd68f-vmflp" Jul 10 02:42:52.042654 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:52.042626 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:42:52.085053 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:52.085022 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxqqk\" (UniqueName: \"kubernetes.io/projected/3110905d-14cd-46a3-9ee0-cda2a2da63aa-kube-api-access-hxqqk\") pod \"maas-api-e2e-mt-b3d83c99-7f5cccd68f-vmflp\" (UID: \"3110905d-14cd-46a3-9ee0-cda2a2da63aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b3d83c99-7f5cccd68f-vmflp" Jul 10 02:42:52.085180 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:52.085108 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3110905d-14cd-46a3-9ee0-cda2a2da63aa-maas-api-tls\") pod \"maas-api-e2e-mt-b3d83c99-7f5cccd68f-vmflp\" (UID: \"3110905d-14cd-46a3-9ee0-cda2a2da63aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b3d83c99-7f5cccd68f-vmflp" Jul 10 02:42:52.085180 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:52.085144 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3110905d-14cd-46a3-9ee0-cda2a2da63aa-openshift-service-ca\") pod \"maas-api-e2e-mt-b3d83c99-7f5cccd68f-vmflp\" (UID: \"3110905d-14cd-46a3-9ee0-cda2a2da63aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b3d83c99-7f5cccd68f-vmflp" Jul 10 02:42:52.085732 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:52.085712 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3110905d-14cd-46a3-9ee0-cda2a2da63aa-openshift-service-ca\") pod \"maas-api-e2e-mt-b3d83c99-7f5cccd68f-vmflp\" (UID: \"3110905d-14cd-46a3-9ee0-cda2a2da63aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b3d83c99-7f5cccd68f-vmflp" Jul 10 02:42:52.087569 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:52.087550 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3110905d-14cd-46a3-9ee0-cda2a2da63aa-maas-api-tls\") pod \"maas-api-e2e-mt-b3d83c99-7f5cccd68f-vmflp\" (UID: \"3110905d-14cd-46a3-9ee0-cda2a2da63aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b3d83c99-7f5cccd68f-vmflp" Jul 10 02:42:52.094639 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:52.094620 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxqqk\" (UniqueName: \"kubernetes.io/projected/3110905d-14cd-46a3-9ee0-cda2a2da63aa-kube-api-access-hxqqk\") pod \"maas-api-e2e-mt-b3d83c99-7f5cccd68f-vmflp\" (UID: \"3110905d-14cd-46a3-9ee0-cda2a2da63aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b3d83c99-7f5cccd68f-vmflp" Jul 10 02:42:52.102422 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:52.102402 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b3d83c99-7f5cccd68f-vmflp" Jul 10 02:42:52.239443 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:52.239413 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b3d83c99-7f5cccd68f-vmflp"] Jul 10 02:42:52.241272 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:42:52.241240 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3110905d_14cd_46a3_9ee0_cda2a2da63aa.slice/crio-8d60f53ba32529bbabbc5a990ed097e721154990db51e13908c9978010d2e848 WatchSource:0}: Error finding container 8d60f53ba32529bbabbc5a990ed097e721154990db51e13908c9978010d2e848: Status 404 returned error can't find the container with id 8d60f53ba32529bbabbc5a990ed097e721154990db51e13908c9978010d2e848 Jul 10 02:42:52.259902 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:52.259878 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b3d83c99-7f5cccd68f-vmflp" event={"ID":"3110905d-14cd-46a3-9ee0-cda2a2da63aa","Type":"ContainerStarted","Data":"8d60f53ba32529bbabbc5a990ed097e721154990db51e13908c9978010d2e848"} Jul 10 02:42:52.406104 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:52.406072 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:42:52.505538 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:52.505461 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:42:53.266482 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:53.266441 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b3d83c99-7f5cccd68f-vmflp" event={"ID":"3110905d-14cd-46a3-9ee0-cda2a2da63aa","Type":"ContainerStarted","Data":"9ed77c57eceedf414b45f7209d1d3527adaa208fcfcf794bc3cc55b47bd4a181"} Jul 10 02:42:53.266922 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:53.266626 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b3d83c99-7f5cccd68f-vmflp" Jul 10 02:42:53.287234 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:53.287184 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b3d83c99-7f5cccd68f-vmflp" podStartSLOduration=1.843090139 podStartE2EDuration="2.287168727s" totalImagesPullingTime="444.078588ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:42:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:42:53.286056571 +0000 UTC m=+2104.862333659" watchObservedRunningTime="2026-07-10 02:42:53.287168727 +0000 UTC m=+2104.863445814" Jul 10 02:42:56.232131 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:56.232102 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:42:59.276524 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:42:59.276494 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b3d83c99-7f5cccd68f-vmflp" Jul 10 02:43:06.475661 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:43:06.475626 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:43:11.836433 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:43:11.836388 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:43:29.488288 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:43:29.488241 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b3d83c99-4rprd"] Jul 10 02:43:29.491922 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:43:29.491906 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b3d83c99-4rprd" Jul 10 02:43:29.494445 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:43:29.494423 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rppvw\"" Jul 10 02:43:29.501016 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:43:29.500996 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b3d83c99-4rprd"] Jul 10 02:43:29.589009 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:43:29.588966 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8f49b08-a63d-4b40-931c-ed5d4d4e74fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b3d83c99-4rprd\" (UID: \"d8f49b08-a63d-4b40-931c-ed5d4d4e74fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b3d83c99-4rprd" Jul 10 02:43:29.690216 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:43:29.690182 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8f49b08-a63d-4b40-931c-ed5d4d4e74fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b3d83c99-4rprd\" (UID: \"d8f49b08-a63d-4b40-931c-ed5d4d4e74fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b3d83c99-4rprd" Jul 10 02:43:29.691119 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:43:29.691098 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8f49b08-a63d-4b40-931c-ed5d4d4e74fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b3d83c99-4rprd\" (UID: \"d8f49b08-a63d-4b40-931c-ed5d4d4e74fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b3d83c99-4rprd" Jul 10 02:43:29.803709 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:43:29.803669 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b3d83c99-4rprd" Jul 10 02:43:29.933663 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:43:29.933635 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b3d83c99-4rprd"] Jul 10 02:43:29.935279 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:43:29.935253 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8f49b08_a63d_4b40_931c_ed5d4d4e74fa.slice/crio-7838ea1b61d5f93a253a410ebdaecf64932c3495641c84ea8a989a014e6da85e WatchSource:0}: Error finding container 7838ea1b61d5f93a253a410ebdaecf64932c3495641c84ea8a989a014e6da85e: Status 404 returned error can't find the container with id 7838ea1b61d5f93a253a410ebdaecf64932c3495641c84ea8a989a014e6da85e Jul 10 02:43:30.472727 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:43:30.472630 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b3d83c99-4rprd" event={"ID":"d8f49b08-a63d-4b40-931c-ed5d4d4e74fa","Type":"ContainerStarted","Data":"1d43ba3245e3e50672b4c1967941e6c9f70be2357fa20f73768b5279905ab5d3"} Jul 10 02:43:30.472727 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:43:30.472687 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b3d83c99-4rprd" event={"ID":"d8f49b08-a63d-4b40-931c-ed5d4d4e74fa","Type":"ContainerStarted","Data":"7838ea1b61d5f93a253a410ebdaecf64932c3495641c84ea8a989a014e6da85e"} Jul 10 02:43:30.489838 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:43:30.489790 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b3d83c99-4rprd" podStartSLOduration=1.489772123 podStartE2EDuration="1.489772123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:43:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:43:30.488099934 +0000 UTC m=+2142.064377021" watchObservedRunningTime="2026-07-10 02:43:30.489772123 +0000 UTC m=+2142.066049209" Jul 10 02:43:31.480058 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:43:31.480023 2566 generic.go:356] "Generic (PLEG): container finished" podID="d8f49b08-a63d-4b40-931c-ed5d4d4e74fa" containerID="1d43ba3245e3e50672b4c1967941e6c9f70be2357fa20f73768b5279905ab5d3" exitCode=0 Jul 10 02:43:31.480058 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:43:31.480059 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b3d83c99-4rprd" event={"ID":"d8f49b08-a63d-4b40-931c-ed5d4d4e74fa","Type":"ContainerDied","Data":"1d43ba3245e3e50672b4c1967941e6c9f70be2357fa20f73768b5279905ab5d3"} Jul 10 02:43:32.618514 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:43:32.618489 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b3d83c99-4rprd" Jul 10 02:43:32.718347 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:43:32.718322 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8f49b08-a63d-4b40-931c-ed5d4d4e74fa-maas-api-service-ca\") pod \"d8f49b08-a63d-4b40-931c-ed5d4d4e74fa\" (UID: \"d8f49b08-a63d-4b40-931c-ed5d4d4e74fa\") " Jul 10 02:43:32.718658 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:43:32.718637 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8f49b08-a63d-4b40-931c-ed5d4d4e74fa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d8f49b08-a63d-4b40-931c-ed5d4d4e74fa" (UID: "d8f49b08-a63d-4b40-931c-ed5d4d4e74fa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 02:43:32.819691 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:43:32.819661 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8f49b08-a63d-4b40-931c-ed5d4d4e74fa-maas-api-service-ca\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:43:33.492526 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:43:33.492497 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b3d83c99-4rprd" Jul 10 02:43:33.492718 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:43:33.492530 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b3d83c99-4rprd" event={"ID":"d8f49b08-a63d-4b40-931c-ed5d4d4e74fa","Type":"ContainerDied","Data":"7838ea1b61d5f93a253a410ebdaecf64932c3495641c84ea8a989a014e6da85e"} Jul 10 02:43:33.492718 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:43:33.492577 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7838ea1b61d5f93a253a410ebdaecf64932c3495641c84ea8a989a014e6da85e" Jul 10 02:43:40.113274 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:43:40.113200 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b3d83c99-7f5cccd68f-vmflp"] Jul 10 02:43:40.113624 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:43:40.113383 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b3d83c99-7f5cccd68f-vmflp" podUID="3110905d-14cd-46a3-9ee0-cda2a2da63aa" containerName="maas-api" containerID="cri-o://9ed77c57eceedf414b45f7209d1d3527adaa208fcfcf794bc3cc55b47bd4a181" gracePeriod=30 Jul 10 02:43:40.135258 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:43:40.135232 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:43:40.363692 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:43:40.363633 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b3d83c99-7f5cccd68f-vmflp" Jul 10 02:43:40.483907 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:43:40.483871 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3110905d-14cd-46a3-9ee0-cda2a2da63aa-maas-api-tls\") pod \"3110905d-14cd-46a3-9ee0-cda2a2da63aa\" (UID: \"3110905d-14cd-46a3-9ee0-cda2a2da63aa\") " Jul 10 02:43:40.484099 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:43:40.483934 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3110905d-14cd-46a3-9ee0-cda2a2da63aa-openshift-service-ca\") pod \"3110905d-14cd-46a3-9ee0-cda2a2da63aa\" (UID: \"3110905d-14cd-46a3-9ee0-cda2a2da63aa\") " Jul 10 02:43:40.484099 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:43:40.483988 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxqqk\" (UniqueName: \"kubernetes.io/projected/3110905d-14cd-46a3-9ee0-cda2a2da63aa-kube-api-access-hxqqk\") pod \"3110905d-14cd-46a3-9ee0-cda2a2da63aa\" (UID: \"3110905d-14cd-46a3-9ee0-cda2a2da63aa\") " Jul 10 02:43:40.484264 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:43:40.484237 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3110905d-14cd-46a3-9ee0-cda2a2da63aa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3110905d-14cd-46a3-9ee0-cda2a2da63aa" (UID: "3110905d-14cd-46a3-9ee0-cda2a2da63aa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 02:43:40.486054 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:43:40.486023 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3110905d-14cd-46a3-9ee0-cda2a2da63aa-kube-api-access-hxqqk" (OuterVolumeSpecName: "kube-api-access-hxqqk") pod "3110905d-14cd-46a3-9ee0-cda2a2da63aa" (UID: "3110905d-14cd-46a3-9ee0-cda2a2da63aa"). InnerVolumeSpecName "kube-api-access-hxqqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 02:43:40.486168 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:43:40.486102 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3110905d-14cd-46a3-9ee0-cda2a2da63aa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3110905d-14cd-46a3-9ee0-cda2a2da63aa" (UID: "3110905d-14cd-46a3-9ee0-cda2a2da63aa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 02:43:40.534028 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:43:40.533997 2566 generic.go:356] "Generic (PLEG): container finished" podID="3110905d-14cd-46a3-9ee0-cda2a2da63aa" containerID="9ed77c57eceedf414b45f7209d1d3527adaa208fcfcf794bc3cc55b47bd4a181" exitCode=0 Jul 10 02:43:40.534161 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:43:40.534041 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b3d83c99-7f5cccd68f-vmflp" Jul 10 02:43:40.534161 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:43:40.534049 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b3d83c99-7f5cccd68f-vmflp" event={"ID":"3110905d-14cd-46a3-9ee0-cda2a2da63aa","Type":"ContainerDied","Data":"9ed77c57eceedf414b45f7209d1d3527adaa208fcfcf794bc3cc55b47bd4a181"} Jul 10 02:43:40.534161 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:43:40.534091 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b3d83c99-7f5cccd68f-vmflp" event={"ID":"3110905d-14cd-46a3-9ee0-cda2a2da63aa","Type":"ContainerDied","Data":"8d60f53ba32529bbabbc5a990ed097e721154990db51e13908c9978010d2e848"} Jul 10 02:43:40.534161 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:43:40.534112 2566 scope.go:117] "RemoveContainer" containerID="9ed77c57eceedf414b45f7209d1d3527adaa208fcfcf794bc3cc55b47bd4a181" Jul 10 02:43:40.543933 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:43:40.543916 2566 scope.go:117] "RemoveContainer" containerID="9ed77c57eceedf414b45f7209d1d3527adaa208fcfcf794bc3cc55b47bd4a181" Jul 10 02:43:40.544194 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:43:40.544174 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ed77c57eceedf414b45f7209d1d3527adaa208fcfcf794bc3cc55b47bd4a181\": container with ID starting with 9ed77c57eceedf414b45f7209d1d3527adaa208fcfcf794bc3cc55b47bd4a181 not found: ID does not exist" containerID="9ed77c57eceedf414b45f7209d1d3527adaa208fcfcf794bc3cc55b47bd4a181" Jul 10 02:43:40.544244 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:43:40.544204 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ed77c57eceedf414b45f7209d1d3527adaa208fcfcf794bc3cc55b47bd4a181"} err="failed to get container status \"9ed77c57eceedf414b45f7209d1d3527adaa208fcfcf794bc3cc55b47bd4a181\": rpc error: code = NotFound desc = could not find container \"9ed77c57eceedf414b45f7209d1d3527adaa208fcfcf794bc3cc55b47bd4a181\": container with ID starting with 9ed77c57eceedf414b45f7209d1d3527adaa208fcfcf794bc3cc55b47bd4a181 not found: ID does not exist" Jul 10 02:43:40.558085 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:43:40.558056 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b3d83c99-7f5cccd68f-vmflp"] Jul 10 02:43:40.563273 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:43:40.563253 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b3d83c99-7f5cccd68f-vmflp"] Jul 10 02:43:40.585199 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:43:40.585180 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3110905d-14cd-46a3-9ee0-cda2a2da63aa-maas-api-tls\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:43:40.585286 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:43:40.585201 2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3110905d-14cd-46a3-9ee0-cda2a2da63aa-openshift-service-ca\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:43:40.585286 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:43:40.585210 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxqqk\" (UniqueName: \"kubernetes.io/projected/3110905d-14cd-46a3-9ee0-cda2a2da63aa-kube-api-access-hxqqk\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:43:41.011296 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:43:41.011267 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3110905d-14cd-46a3-9ee0-cda2a2da63aa" path="/var/lib/kubelet/pods/3110905d-14cd-46a3-9ee0-cda2a2da63aa/volumes" Jul 10 02:43:44.594817 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:43:44.594783 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:43:49.083680 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:43:49.083651 2566 scope.go:117] "RemoveContainer" containerID="bfbd0e6f7d27032eb53b55f494c17df68126e40ec2512df16717d7164d382f38" Jul 10 02:43:49.621770 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:43:49.621737 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:44:11.378335 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:44:11.378299 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:44:11.478801 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:44:11.478761 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:44:26.359806 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:44:26.359772 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:44:27.194157 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:44:27.194122 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:44:32.895476 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:44:32.895431 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:44:32.954929 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:44:32.954892 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:44:36.799911 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:44:36.799875 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:44:37.030873 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:44:37.030841 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:44:38.125948 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:44:38.125900 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:44:38.224214 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:44:38.224180 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:44:38.324572 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:44:38.324537 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:44:39.146356 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:44:39.146319 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:44:39.241778 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:44:39.241746 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:44:39.342092 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:44:39.342061 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:44:40.789978 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:44:40.789948 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:44:47.676529 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:44:47.676498 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:44:47.777440 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:44:47.777397 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:44:48.384974 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:44:48.384944 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:44:54.569688 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:44:54.569642 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:44:54.639349 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:44:54.639310 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:44:54.724114 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:44:54.724084 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:44:54.822799 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:44:54.822704 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:45:00.146530 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:00.146501 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727525-9gdn7"] Jul 10 02:45:00.147066 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:00.146894 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3110905d-14cd-46a3-9ee0-cda2a2da63aa" containerName="maas-api" Jul 10 02:45:00.147066 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:00.146908 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3110905d-14cd-46a3-9ee0-cda2a2da63aa" containerName="maas-api" Jul 10 02:45:00.147066 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:00.146932 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8f49b08-a63d-4b40-931c-ed5d4d4e74fa" containerName="revoke-keys" Jul 10 02:45:00.147066 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:00.146940 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8f49b08-a63d-4b40-931c-ed5d4d4e74fa" containerName="revoke-keys" Jul 10 02:45:00.147066 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:00.147035 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3110905d-14cd-46a3-9ee0-cda2a2da63aa" containerName="maas-api" Jul 10 02:45:00.147066 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:00.147047 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8f49b08-a63d-4b40-931c-ed5d4d4e74fa" containerName="revoke-keys" Jul 10 02:45:00.150909 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:00.150885 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727525-9gdn7" Jul 10 02:45:00.154038 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:00.153501 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rppvw\"" Jul 10 02:45:00.155614 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:00.155575 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727525-9gdn7"] Jul 10 02:45:00.164471 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:00.164453 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727525-9gdn7" Jul 10 02:45:00.299228 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:00.299200 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727525-9gdn7"] Jul 10 02:45:00.301193 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:45:00.301169 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod030df5fa_92a4_45fe_8e7f_32da38cad42f.slice/crio-c975b301ed7073fb00aa18d498ae67ad3569735248d83b4b55c00109e9d04896 WatchSource:0}: Error finding container c975b301ed7073fb00aa18d498ae67ad3569735248d83b4b55c00109e9d04896: Status 404 returned error can't find the container with id c975b301ed7073fb00aa18d498ae67ad3569735248d83b4b55c00109e9d04896 Jul 10 02:45:00.982445 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:00.982408 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727525-9gdn7" event={"ID":"030df5fa-92a4-45fe-8e7f-32da38cad42f","Type":"ContainerStarted","Data":"ea2b3f0eb2a090e9cc98f5ab9da0d94b7393ae05787c25f02845584b15666780"} Jul 10 02:45:00.982684 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:00.982466 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727525-9gdn7" event={"ID":"030df5fa-92a4-45fe-8e7f-32da38cad42f","Type":"ContainerStarted","Data":"c975b301ed7073fb00aa18d498ae67ad3569735248d83b4b55c00109e9d04896"} Jul 10 02:45:01.001005 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:01.000953 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727525-9gdn7" podStartSLOduration=1.000936552 podStartE2EDuration="1.000936552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:45:00.998115922 +0000 UTC m=+2232.574393005" watchObservedRunningTime="2026-07-10 02:45:01.000936552 +0000 UTC m=+2232.577213638" Jul 10 02:45:01.989921 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:01.989882 2566 generic.go:356] "Generic (PLEG): container finished" podID="030df5fa-92a4-45fe-8e7f-32da38cad42f" containerID="ea2b3f0eb2a090e9cc98f5ab9da0d94b7393ae05787c25f02845584b15666780" exitCode=0 Jul 10 02:45:01.990328 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:01.989929 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727525-9gdn7" event={"ID":"030df5fa-92a4-45fe-8e7f-32da38cad42f","Type":"ContainerDied","Data":"ea2b3f0eb2a090e9cc98f5ab9da0d94b7393ae05787c25f02845584b15666780"} Jul 10 02:45:03.133160 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:03.133136 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727525-9gdn7" Jul 10 02:45:04.001788 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:04.001755 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727525-9gdn7" Jul 10 02:45:04.001788 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:04.001779 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727525-9gdn7" event={"ID":"030df5fa-92a4-45fe-8e7f-32da38cad42f","Type":"ContainerDied","Data":"c975b301ed7073fb00aa18d498ae67ad3569735248d83b4b55c00109e9d04896"} Jul 10 02:45:04.002001 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:04.001844 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c975b301ed7073fb00aa18d498ae67ad3569735248d83b4b55c00109e9d04896" Jul 10 02:45:05.323149 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:05.323115 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:45:05.714564 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:05.714487 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:45:26.709458 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:26.709414 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:45:32.866220 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:32.866187 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:45:32.966257 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:32.966230 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:45:33.266329 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:33.266247 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:45:33.625970 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:33.625935 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:45:33.803120 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:33.803086 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:45:36.559098 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:36.559057 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:45:36.659473 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:36.659441 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:45:36.767744 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:36.767713 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d76849-cc56b547b-twnf5"] Jul 10 02:45:36.768172 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:36.768154 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="030df5fa-92a4-45fe-8e7f-32da38cad42f" containerName="cleanup" Jul 10 02:45:36.768273 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:36.768174 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="030df5fa-92a4-45fe-8e7f-32da38cad42f" containerName="cleanup" Jul 10 02:45:36.768327 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:36.768290 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="030df5fa-92a4-45fe-8e7f-32da38cad42f" containerName="cleanup" Jul 10 02:45:36.773412 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:36.773389 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d76849-cc56b547b-twnf5" Jul 10 02:45:36.776826 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:36.776802 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 02:45:36.776941 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:36.776827 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-d76849-serving-cert\"" Jul 10 02:45:36.776941 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:36.776837 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 10 02:45:36.776941 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:36.776839 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-r7s9m\"" Jul 10 02:45:36.782713 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:36.782693 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d76849-cc56b547b-twnf5"] Jul 10 02:45:36.811801 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:36.811714 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:45:36.858951 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:36.858912 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6518692-ddba-4aa1-b847-01520b11d3e9-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d76849-cc56b547b-twnf5\" (UID: \"a6518692-ddba-4aa1-b847-01520b11d3e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d76849-cc56b547b-twnf5" Jul 10 02:45:36.859273 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:36.859028 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6518692-ddba-4aa1-b847-01520b11d3e9-maas-api-tls\") pod \"maas-api-e2e-inf-a-d76849-cc56b547b-twnf5\" (UID: \"a6518692-ddba-4aa1-b847-01520b11d3e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d76849-cc56b547b-twnf5" Jul 10 02:45:36.859273 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:36.859142 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfmjx\" (UniqueName: \"kubernetes.io/projected/a6518692-ddba-4aa1-b847-01520b11d3e9-kube-api-access-nfmjx\") pod \"maas-api-e2e-inf-a-d76849-cc56b547b-twnf5\" (UID: \"a6518692-ddba-4aa1-b847-01520b11d3e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d76849-cc56b547b-twnf5" Jul 10 02:45:36.960195 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:36.960163 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6518692-ddba-4aa1-b847-01520b11d3e9-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d76849-cc56b547b-twnf5\" (UID: \"a6518692-ddba-4aa1-b847-01520b11d3e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d76849-cc56b547b-twnf5" Jul 10 02:45:36.960318 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:36.960209 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6518692-ddba-4aa1-b847-01520b11d3e9-maas-api-tls\") pod \"maas-api-e2e-inf-a-d76849-cc56b547b-twnf5\" (UID: \"a6518692-ddba-4aa1-b847-01520b11d3e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d76849-cc56b547b-twnf5" Jul 10 02:45:36.960318 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:36.960261 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfmjx\" (UniqueName: \"kubernetes.io/projected/a6518692-ddba-4aa1-b847-01520b11d3e9-kube-api-access-nfmjx\") pod \"maas-api-e2e-inf-a-d76849-cc56b547b-twnf5\" (UID: \"a6518692-ddba-4aa1-b847-01520b11d3e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d76849-cc56b547b-twnf5" Jul 10 02:45:36.960881 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:36.960848 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6518692-ddba-4aa1-b847-01520b11d3e9-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d76849-cc56b547b-twnf5\" (UID: \"a6518692-ddba-4aa1-b847-01520b11d3e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d76849-cc56b547b-twnf5" Jul 10 02:45:36.962683 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:36.962665 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6518692-ddba-4aa1-b847-01520b11d3e9-maas-api-tls\") pod \"maas-api-e2e-inf-a-d76849-cc56b547b-twnf5\" (UID: \"a6518692-ddba-4aa1-b847-01520b11d3e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d76849-cc56b547b-twnf5" Jul 10 02:45:36.971908 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:36.971887 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfmjx\" (UniqueName: \"kubernetes.io/projected/a6518692-ddba-4aa1-b847-01520b11d3e9-kube-api-access-nfmjx\") pod \"maas-api-e2e-inf-a-d76849-cc56b547b-twnf5\" (UID: \"a6518692-ddba-4aa1-b847-01520b11d3e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d76849-cc56b547b-twnf5" Jul 10 02:45:37.037903 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:37.037872 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:45:37.085565 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:37.085500 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d76849-cc56b547b-twnf5" Jul 10 02:45:37.223302 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:37.223279 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d76849-cc56b547b-twnf5"] Jul 10 02:45:37.225392 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:45:37.225360 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6518692_ddba_4aa1_b847_01520b11d3e9.slice/crio-73f108904d6184e5c6700031ccf4ba699249c180d4a3b17e896fc32b9ca5a1fc WatchSource:0}: Error finding container 73f108904d6184e5c6700031ccf4ba699249c180d4a3b17e896fc32b9ca5a1fc: Status 404 returned error can't find the container with id 73f108904d6184e5c6700031ccf4ba699249c180d4a3b17e896fc32b9ca5a1fc Jul 10 02:45:37.227111 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:37.227089 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 02:45:37.341958 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:37.341870 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:45:37.442279 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:37.442248 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:45:38.207680 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:38.207632 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d76849-cc56b547b-twnf5" event={"ID":"a6518692-ddba-4aa1-b847-01520b11d3e9","Type":"ContainerStarted","Data":"60d6abb734401bb6c81885744c90fa029e79f9649480f3880615e65f55c7bd22"} Jul 10 02:45:38.208134 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:38.207728 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d76849-cc56b547b-twnf5" event={"ID":"a6518692-ddba-4aa1-b847-01520b11d3e9","Type":"ContainerStarted","Data":"73f108904d6184e5c6700031ccf4ba699249c180d4a3b17e896fc32b9ca5a1fc"} Jul 10 02:45:38.208134 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:38.207749 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d76849-cc56b547b-twnf5" Jul 10 02:45:38.227055 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:38.227007 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d76849-cc56b547b-twnf5" podStartSLOduration=1.781301784 podStartE2EDuration="2.226993159s" totalImagesPullingTime="445.691375ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:45:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:45:38.224955153 +0000 UTC m=+2269.801232241" watchObservedRunningTime="2026-07-10 02:45:38.226993159 +0000 UTC m=+2269.803270280" Jul 10 02:45:41.423660 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:41.423627 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:45:41.911420 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:41.911391 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:45:42.274125 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:42.274088 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:45:42.461661 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:42.461621 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:45:44.218616 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:44.218565 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d76849-cc56b547b-twnf5" Jul 10 02:45:45.652518 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:45.652480 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:45:45.746741 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:45.746708 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bbcae0-bb867d558-z4xcs"] Jul 10 02:45:45.751269 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:45.751240 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bbcae0-bb867d558-z4xcs" Jul 10 02:45:45.754811 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:45.754784 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-bbcae0-serving-cert\"" Jul 10 02:45:45.761554 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:45.761528 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bbcae0-bb867d558-z4xcs"] Jul 10 02:45:45.848294 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:45.848262 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15c6bb65-87c0-4413-b15f-90938f0cea47-maas-api-tls\") pod \"maas-api-e2e-inf-b-bbcae0-bb867d558-z4xcs\" (UID: \"15c6bb65-87c0-4413-b15f-90938f0cea47\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bbcae0-bb867d558-z4xcs" Jul 10 02:45:45.848467 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:45.848300 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15c6bb65-87c0-4413-b15f-90938f0cea47-openshift-service-ca\") pod \"maas-api-e2e-inf-b-bbcae0-bb867d558-z4xcs\" (UID: \"15c6bb65-87c0-4413-b15f-90938f0cea47\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bbcae0-bb867d558-z4xcs" Jul 10 02:45:45.848467 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:45.848361 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfjgh\" (UniqueName: \"kubernetes.io/projected/15c6bb65-87c0-4413-b15f-90938f0cea47-kube-api-access-hfjgh\") pod \"maas-api-e2e-inf-b-bbcae0-bb867d558-z4xcs\" (UID: \"15c6bb65-87c0-4413-b15f-90938f0cea47\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bbcae0-bb867d558-z4xcs" Jul 10 02:45:45.949609 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:45.949503 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfjgh\" (UniqueName: \"kubernetes.io/projected/15c6bb65-87c0-4413-b15f-90938f0cea47-kube-api-access-hfjgh\") pod \"maas-api-e2e-inf-b-bbcae0-bb867d558-z4xcs\" (UID: \"15c6bb65-87c0-4413-b15f-90938f0cea47\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bbcae0-bb867d558-z4xcs" Jul 10 02:45:45.949788 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:45.949622 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15c6bb65-87c0-4413-b15f-90938f0cea47-maas-api-tls\") pod \"maas-api-e2e-inf-b-bbcae0-bb867d558-z4xcs\" (UID: \"15c6bb65-87c0-4413-b15f-90938f0cea47\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bbcae0-bb867d558-z4xcs" Jul 10 02:45:45.949788 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:45.949656 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15c6bb65-87c0-4413-b15f-90938f0cea47-openshift-service-ca\") pod \"maas-api-e2e-inf-b-bbcae0-bb867d558-z4xcs\" (UID: \"15c6bb65-87c0-4413-b15f-90938f0cea47\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bbcae0-bb867d558-z4xcs" Jul 10 02:45:45.949788 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:45:45.949768 2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-bbcae0-serving-cert: secret "maas-api-e2e-inf-b-bbcae0-serving-cert" not found Jul 10 02:45:45.949998 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:45:45.949835 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15c6bb65-87c0-4413-b15f-90938f0cea47-maas-api-tls podName:15c6bb65-87c0-4413-b15f-90938f0cea47 nodeName:}" failed. No retries permitted until 2026-07-10 02:45:46.44981644 +0000 UTC m=+2278.026093506 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/15c6bb65-87c0-4413-b15f-90938f0cea47-maas-api-tls") pod "maas-api-e2e-inf-b-bbcae0-bb867d558-z4xcs" (UID: "15c6bb65-87c0-4413-b15f-90938f0cea47") : secret "maas-api-e2e-inf-b-bbcae0-serving-cert" not found Jul 10 02:45:45.950348 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:45.950325 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15c6bb65-87c0-4413-b15f-90938f0cea47-openshift-service-ca\") pod \"maas-api-e2e-inf-b-bbcae0-bb867d558-z4xcs\" (UID: \"15c6bb65-87c0-4413-b15f-90938f0cea47\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bbcae0-bb867d558-z4xcs" Jul 10 02:45:45.961805 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:45.961777 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfjgh\" (UniqueName: \"kubernetes.io/projected/15c6bb65-87c0-4413-b15f-90938f0cea47-kube-api-access-hfjgh\") pod \"maas-api-e2e-inf-b-bbcae0-bb867d558-z4xcs\" (UID: \"15c6bb65-87c0-4413-b15f-90938f0cea47\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bbcae0-bb867d558-z4xcs" Jul 10 02:45:46.454078 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:46.454042 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15c6bb65-87c0-4413-b15f-90938f0cea47-maas-api-tls\") pod \"maas-api-e2e-inf-b-bbcae0-bb867d558-z4xcs\" (UID: \"15c6bb65-87c0-4413-b15f-90938f0cea47\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bbcae0-bb867d558-z4xcs" Jul 10 02:45:46.456463 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:46.456433 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15c6bb65-87c0-4413-b15f-90938f0cea47-maas-api-tls\") pod \"maas-api-e2e-inf-b-bbcae0-bb867d558-z4xcs\" (UID: \"15c6bb65-87c0-4413-b15f-90938f0cea47\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bbcae0-bb867d558-z4xcs" Jul 10 02:45:46.669537 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:46.669498 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bbcae0-bb867d558-z4xcs" Jul 10 02:45:46.807082 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:46.807054 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bbcae0-bb867d558-z4xcs"] Jul 10 02:45:46.808821 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:45:46.808786 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15c6bb65_87c0_4413_b15f_90938f0cea47.slice/crio-fdfc6dd6b080f6d8c7665fe3daab294793e4b5367d7ec6c72614f36de7bc8584 WatchSource:0}: Error finding container fdfc6dd6b080f6d8c7665fe3daab294793e4b5367d7ec6c72614f36de7bc8584: Status 404 returned error can't find the container with id fdfc6dd6b080f6d8c7665fe3daab294793e4b5367d7ec6c72614f36de7bc8584 Jul 10 02:45:46.815327 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:46.815305 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:45:47.258850 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:47.258824 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bbcae0-bb867d558-z4xcs" event={"ID":"15c6bb65-87c0-4413-b15f-90938f0cea47","Type":"ContainerStarted","Data":"f17e5a335c718fdc6787877ac42452f8ee25ae399ab85bf3991e5f05bb37757f"} Jul 10 02:45:47.258966 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:47.258869 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bbcae0-bb867d558-z4xcs" event={"ID":"15c6bb65-87c0-4413-b15f-90938f0cea47","Type":"ContainerStarted","Data":"fdfc6dd6b080f6d8c7665fe3daab294793e4b5367d7ec6c72614f36de7bc8584"} Jul 10 02:45:47.258966 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:47.258938 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bbcae0-bb867d558-z4xcs" Jul 10 02:45:47.278210 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:47.278162 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bbcae0-bb867d558-z4xcs" podStartSLOduration=1.902075899 podStartE2EDuration="2.278146114s" totalImagesPullingTime="376.070215ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:45:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:45:47.276124539 +0000 UTC m=+2278.852401638" watchObservedRunningTime="2026-07-10 02:45:47.278146114 +0000 UTC m=+2278.854423199" Jul 10 02:45:49.115532 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:49.115498 2566 scope.go:117] "RemoveContainer" containerID="ae76133b1e5dd5ff8325e2e74eace1ac6fad990757320ffbf9ba9560aacab2b1" Jul 10 02:45:50.273084 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:50.273029 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:45:53.269795 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:53.269769 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bbcae0-bb867d558-z4xcs" Jul 10 02:45:59.149660 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:59.149622 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:45:59.246378 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:45:59.246345 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:46:07.339574 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:46:07.339542 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:46:07.449328 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:46:07.449298 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:46:12.771315 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:46:12.771284 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:46:17.451553 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:46:17.451521 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:46:28.571209 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:46:28.571175 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:46:37.982694 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:46:37.982652 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:46:59.199975 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:46:59.199940 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:47:00.109187 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:00.109151 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:47:00.240122 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:00.240091 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d76849-7m772"] Jul 10 02:47:00.243662 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:00.243645 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d76849-7m772" Jul 10 02:47:00.246076 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:00.246055 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rppvw\"" Jul 10 02:47:00.250986 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:00.250925 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d76849-7m772"] Jul 10 02:47:00.274396 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:00.274374 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d88b769e-b5d5-4e09-8f37-ebdd1efdd639-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d76849-7m772\" (UID: \"d88b769e-b5d5-4e09-8f37-ebdd1efdd639\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d76849-7m772" Jul 10 02:47:00.374798 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:00.374738 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d88b769e-b5d5-4e09-8f37-ebdd1efdd639-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d76849-7m772\" (UID: \"d88b769e-b5d5-4e09-8f37-ebdd1efdd639\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d76849-7m772" Jul 10 02:47:00.375293 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:00.375275 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d88b769e-b5d5-4e09-8f37-ebdd1efdd639-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d76849-7m772\" (UID: \"d88b769e-b5d5-4e09-8f37-ebdd1efdd639\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d76849-7m772" Jul 10 02:47:00.556711 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:00.556677 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d76849-7m772" Jul 10 02:47:00.691033 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:00.690879 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d76849-7m772"] Jul 10 02:47:00.694409 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:47:00.694381 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd88b769e_b5d5_4e09_8f37_ebdd1efdd639.slice/crio-99dc97aa134cf40e789d8806d4c4c1067f996f37021d26916a0ca8df101fa38f WatchSource:0}: Error finding container 99dc97aa134cf40e789d8806d4c4c1067f996f37021d26916a0ca8df101fa38f: Status 404 returned error can't find the container with id 99dc97aa134cf40e789d8806d4c4c1067f996f37021d26916a0ca8df101fa38f Jul 10 02:47:00.894226 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:00.894156 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:47:00.994169 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:00.994134 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:47:01.677889 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:01.677849 2566 generic.go:356] "Generic (PLEG): container finished" podID="d88b769e-b5d5-4e09-8f37-ebdd1efdd639" containerID="68c19f7c0cae4080ef3e2a51b655543b16643fbaa7cf5ab58dda91ad63df3c4e" exitCode=0 Jul 10 02:47:01.678257 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:01.677933 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d76849-7m772" event={"ID":"d88b769e-b5d5-4e09-8f37-ebdd1efdd639","Type":"ContainerDied","Data":"68c19f7c0cae4080ef3e2a51b655543b16643fbaa7cf5ab58dda91ad63df3c4e"} Jul 10 02:47:01.678257 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:01.678010 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d76849-7m772" event={"ID":"d88b769e-b5d5-4e09-8f37-ebdd1efdd639","Type":"ContainerStarted","Data":"99dc97aa134cf40e789d8806d4c4c1067f996f37021d26916a0ca8df101fa38f"} Jul 10 02:47:02.819764 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:02.819739 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d76849-7m772" Jul 10 02:47:02.896841 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:02.896806 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d88b769e-b5d5-4e09-8f37-ebdd1efdd639-maas-api-service-ca\") pod \"d88b769e-b5d5-4e09-8f37-ebdd1efdd639\" (UID: \"d88b769e-b5d5-4e09-8f37-ebdd1efdd639\") " Jul 10 02:47:02.897201 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:02.897173 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d88b769e-b5d5-4e09-8f37-ebdd1efdd639-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d88b769e-b5d5-4e09-8f37-ebdd1efdd639" (UID: "d88b769e-b5d5-4e09-8f37-ebdd1efdd639"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 02:47:02.998362 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:02.998263 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d88b769e-b5d5-4e09-8f37-ebdd1efdd639-maas-api-service-ca\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:47:03.691144 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:03.691110 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d76849-7m772" Jul 10 02:47:03.691325 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:03.691144 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d76849-7m772" event={"ID":"d88b769e-b5d5-4e09-8f37-ebdd1efdd639","Type":"ContainerDied","Data":"99dc97aa134cf40e789d8806d4c4c1067f996f37021d26916a0ca8df101fa38f"} Jul 10 02:47:03.691325 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:03.691194 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99dc97aa134cf40e789d8806d4c4c1067f996f37021d26916a0ca8df101fa38f" Jul 10 02:47:10.735165 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:10.735130 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d76849-cc56b547b-twnf5"] Jul 10 02:47:10.735584 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:10.735383 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d76849-cc56b547b-twnf5" podUID="a6518692-ddba-4aa1-b847-01520b11d3e9" containerName="maas-api" containerID="cri-o://60d6abb734401bb6c81885744c90fa029e79f9649480f3880615e65f55c7bd22" gracePeriod=30 Jul 10 02:47:10.792759 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:10.792730 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:47:10.994129 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:10.994066 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d76849-cc56b547b-twnf5" Jul 10 02:47:11.069158 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:11.069134 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6518692-ddba-4aa1-b847-01520b11d3e9-maas-api-tls\") pod \"a6518692-ddba-4aa1-b847-01520b11d3e9\" (UID: \"a6518692-ddba-4aa1-b847-01520b11d3e9\") " Jul 10 02:47:11.069287 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:11.069166 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfmjx\" (UniqueName: \"kubernetes.io/projected/a6518692-ddba-4aa1-b847-01520b11d3e9-kube-api-access-nfmjx\") pod \"a6518692-ddba-4aa1-b847-01520b11d3e9\" (UID: \"a6518692-ddba-4aa1-b847-01520b11d3e9\") " Jul 10 02:47:11.069287 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:11.069261 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6518692-ddba-4aa1-b847-01520b11d3e9-openshift-service-ca\") pod \"a6518692-ddba-4aa1-b847-01520b11d3e9\" (UID: \"a6518692-ddba-4aa1-b847-01520b11d3e9\") " Jul 10 02:47:11.069585 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:11.069557 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6518692-ddba-4aa1-b847-01520b11d3e9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a6518692-ddba-4aa1-b847-01520b11d3e9" (UID: "a6518692-ddba-4aa1-b847-01520b11d3e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 02:47:11.071181 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:11.071154 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6518692-ddba-4aa1-b847-01520b11d3e9-kube-api-access-nfmjx" (OuterVolumeSpecName: "kube-api-access-nfmjx") pod "a6518692-ddba-4aa1-b847-01520b11d3e9" (UID: "a6518692-ddba-4aa1-b847-01520b11d3e9"). InnerVolumeSpecName "kube-api-access-nfmjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 02:47:11.071288 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:11.071231 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6518692-ddba-4aa1-b847-01520b11d3e9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a6518692-ddba-4aa1-b847-01520b11d3e9" (UID: "a6518692-ddba-4aa1-b847-01520b11d3e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 02:47:11.170097 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:11.170071 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6518692-ddba-4aa1-b847-01520b11d3e9-maas-api-tls\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:47:11.170097 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:11.170095 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfmjx\" (UniqueName: \"kubernetes.io/projected/a6518692-ddba-4aa1-b847-01520b11d3e9-kube-api-access-nfmjx\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:47:11.170260 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:11.170110 2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6518692-ddba-4aa1-b847-01520b11d3e9-openshift-service-ca\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:47:11.736244 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:11.736211 2566 generic.go:356] "Generic (PLEG): container finished" podID="a6518692-ddba-4aa1-b847-01520b11d3e9" containerID="60d6abb734401bb6c81885744c90fa029e79f9649480f3880615e65f55c7bd22" exitCode=0 Jul 10 02:47:11.736696 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:11.736257 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d76849-cc56b547b-twnf5" Jul 10 02:47:11.736696 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:11.736299 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d76849-cc56b547b-twnf5" event={"ID":"a6518692-ddba-4aa1-b847-01520b11d3e9","Type":"ContainerDied","Data":"60d6abb734401bb6c81885744c90fa029e79f9649480f3880615e65f55c7bd22"} Jul 10 02:47:11.736696 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:11.736351 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d76849-cc56b547b-twnf5" event={"ID":"a6518692-ddba-4aa1-b847-01520b11d3e9","Type":"ContainerDied","Data":"73f108904d6184e5c6700031ccf4ba699249c180d4a3b17e896fc32b9ca5a1fc"} Jul 10 02:47:11.736696 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:11.736376 2566 scope.go:117] "RemoveContainer" containerID="60d6abb734401bb6c81885744c90fa029e79f9649480f3880615e65f55c7bd22" Jul 10 02:47:11.746503 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:11.746490 2566 scope.go:117] "RemoveContainer" containerID="60d6abb734401bb6c81885744c90fa029e79f9649480f3880615e65f55c7bd22" Jul 10 02:47:11.746772 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:47:11.746754 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60d6abb734401bb6c81885744c90fa029e79f9649480f3880615e65f55c7bd22\": container with ID starting with 60d6abb734401bb6c81885744c90fa029e79f9649480f3880615e65f55c7bd22 not found: ID does not exist" containerID="60d6abb734401bb6c81885744c90fa029e79f9649480f3880615e65f55c7bd22" Jul 10 02:47:11.746880 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:11.746779 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60d6abb734401bb6c81885744c90fa029e79f9649480f3880615e65f55c7bd22"} err="failed to get container status \"60d6abb734401bb6c81885744c90fa029e79f9649480f3880615e65f55c7bd22\": rpc error: code = NotFound desc = could not find container \"60d6abb734401bb6c81885744c90fa029e79f9649480f3880615e65f55c7bd22\": container with ID starting with 60d6abb734401bb6c81885744c90fa029e79f9649480f3880615e65f55c7bd22 not found: ID does not exist" Jul 10 02:47:11.760872 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:11.760850 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d76849-cc56b547b-twnf5"] Jul 10 02:47:11.765574 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:11.765550 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d76849-cc56b547b-twnf5"] Jul 10 02:47:13.011768 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:13.011732 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6518692-ddba-4aa1-b847-01520b11d3e9" path="/var/lib/kubelet/pods/a6518692-ddba-4aa1-b847-01520b11d3e9/volumes" Jul 10 02:47:15.344369 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:15.344338 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:47:21.506577 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:21.506530 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:47:21.606309 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:21.606263 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:47:26.087125 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:26.087098 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:47:26.331865 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:26.331822 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:47:26.466258 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:26.466181 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bbcae0-nc5mg"] Jul 10 02:47:26.467131 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:26.467101 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6518692-ddba-4aa1-b847-01520b11d3e9" containerName="maas-api" Jul 10 02:47:26.467131 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:26.467131 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6518692-ddba-4aa1-b847-01520b11d3e9" containerName="maas-api" Jul 10 02:47:26.467320 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:26.467149 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d88b769e-b5d5-4e09-8f37-ebdd1efdd639" containerName="revoke-keys" Jul 10 02:47:26.467320 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:26.467167 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d88b769e-b5d5-4e09-8f37-ebdd1efdd639" containerName="revoke-keys" Jul 10 02:47:26.467479 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:26.467464 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6518692-ddba-4aa1-b847-01520b11d3e9" containerName="maas-api" Jul 10 02:47:26.467535 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:26.467496 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d88b769e-b5d5-4e09-8f37-ebdd1efdd639" containerName="revoke-keys" Jul 10 02:47:26.472330 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:26.472304 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bbcae0-nc5mg" Jul 10 02:47:26.475801 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:26.475775 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rppvw\"" Jul 10 02:47:26.477919 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:26.477895 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bbcae0-nc5mg"] Jul 10 02:47:26.610026 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:26.609991 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d97d4969-28ef-4159-a3a8-20d852a3255b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bbcae0-nc5mg\" (UID: \"d97d4969-28ef-4159-a3a8-20d852a3255b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bbcae0-nc5mg" Jul 10 02:47:26.710826 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:26.710797 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d97d4969-28ef-4159-a3a8-20d852a3255b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bbcae0-nc5mg\" (UID: \"d97d4969-28ef-4159-a3a8-20d852a3255b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bbcae0-nc5mg" Jul 10 02:47:26.711381 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:26.711362 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d97d4969-28ef-4159-a3a8-20d852a3255b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bbcae0-nc5mg\" (UID: \"d97d4969-28ef-4159-a3a8-20d852a3255b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bbcae0-nc5mg" Jul 10 02:47:26.786538 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:26.786514 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bbcae0-nc5mg" Jul 10 02:47:26.928159 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:26.928131 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bbcae0-nc5mg"] Jul 10 02:47:26.929617 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:47:26.929560 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd97d4969_28ef_4159_a3a8_20d852a3255b.slice/crio-f642bd2d25b0668c9338ca42bfe3f1176b22cff84207b28d1e752b3ee4cc4c26 WatchSource:0}: Error finding container f642bd2d25b0668c9338ca42bfe3f1176b22cff84207b28d1e752b3ee4cc4c26: Status 404 returned error can't find the container with id f642bd2d25b0668c9338ca42bfe3f1176b22cff84207b28d1e752b3ee4cc4c26 Jul 10 02:47:27.128678 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:27.128652 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:47:27.214249 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:27.214221 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:47:27.832986 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:27.832951 2566 generic.go:356] "Generic (PLEG): container finished" podID="d97d4969-28ef-4159-a3a8-20d852a3255b" containerID="cf65dc318fad4d09b596e678cc69eb0e130e72bc26d6c3b62d633a93aecba9ba" exitCode=0 Jul 10 02:47:27.833221 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:27.833000 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bbcae0-nc5mg" event={"ID":"d97d4969-28ef-4159-a3a8-20d852a3255b","Type":"ContainerDied","Data":"cf65dc318fad4d09b596e678cc69eb0e130e72bc26d6c3b62d633a93aecba9ba"} Jul 10 02:47:27.833221 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:27.833091 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bbcae0-nc5mg" event={"ID":"d97d4969-28ef-4159-a3a8-20d852a3255b","Type":"ContainerStarted","Data":"f642bd2d25b0668c9338ca42bfe3f1176b22cff84207b28d1e752b3ee4cc4c26"} Jul 10 02:47:28.965925 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:28.965902 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bbcae0-nc5mg" Jul 10 02:47:29.129829 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:29.129760 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d97d4969-28ef-4159-a3a8-20d852a3255b-maas-api-service-ca\") pod \"d97d4969-28ef-4159-a3a8-20d852a3255b\" (UID: \"d97d4969-28ef-4159-a3a8-20d852a3255b\") " Jul 10 02:47:29.130121 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:29.130088 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d97d4969-28ef-4159-a3a8-20d852a3255b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d97d4969-28ef-4159-a3a8-20d852a3255b" (UID: "d97d4969-28ef-4159-a3a8-20d852a3255b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 02:47:29.231020 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:29.230994 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d97d4969-28ef-4159-a3a8-20d852a3255b-maas-api-service-ca\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:47:29.845932 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:29.845903 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bbcae0-nc5mg" Jul 10 02:47:29.846121 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:29.845899 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bbcae0-nc5mg" event={"ID":"d97d4969-28ef-4159-a3a8-20d852a3255b","Type":"ContainerDied","Data":"f642bd2d25b0668c9338ca42bfe3f1176b22cff84207b28d1e752b3ee4cc4c26"} Jul 10 02:47:29.846121 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:29.846025 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f642bd2d25b0668c9338ca42bfe3f1176b22cff84207b28d1e752b3ee4cc4c26" Jul 10 02:47:36.962379 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:36.962336 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bbcae0-bb867d558-z4xcs"] Jul 10 02:47:36.962867 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:36.962572 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bbcae0-bb867d558-z4xcs" podUID="15c6bb65-87c0-4413-b15f-90938f0cea47" containerName="maas-api" containerID="cri-o://f17e5a335c718fdc6787877ac42452f8ee25ae399ab85bf3991e5f05bb37757f" gracePeriod=30 Jul 10 02:47:36.986425 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:36.986396 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:47:37.209818 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:37.209795 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bbcae0-bb867d558-z4xcs" Jul 10 02:47:37.284319 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:37.284294 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15c6bb65-87c0-4413-b15f-90938f0cea47-maas-api-tls\") pod \"15c6bb65-87c0-4413-b15f-90938f0cea47\" (UID: \"15c6bb65-87c0-4413-b15f-90938f0cea47\") " Jul 10 02:47:37.284450 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:37.284359 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfjgh\" (UniqueName: \"kubernetes.io/projected/15c6bb65-87c0-4413-b15f-90938f0cea47-kube-api-access-hfjgh\") pod \"15c6bb65-87c0-4413-b15f-90938f0cea47\" (UID: \"15c6bb65-87c0-4413-b15f-90938f0cea47\") " Jul 10 02:47:37.284450 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:37.284393 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15c6bb65-87c0-4413-b15f-90938f0cea47-openshift-service-ca\") pod \"15c6bb65-87c0-4413-b15f-90938f0cea47\" (UID: \"15c6bb65-87c0-4413-b15f-90938f0cea47\") " Jul 10 02:47:37.284817 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:37.284794 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15c6bb65-87c0-4413-b15f-90938f0cea47-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "15c6bb65-87c0-4413-b15f-90938f0cea47" (UID: "15c6bb65-87c0-4413-b15f-90938f0cea47"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 02:47:37.286347 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:37.286325 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15c6bb65-87c0-4413-b15f-90938f0cea47-kube-api-access-hfjgh" (OuterVolumeSpecName: "kube-api-access-hfjgh") pod "15c6bb65-87c0-4413-b15f-90938f0cea47" (UID: "15c6bb65-87c0-4413-b15f-90938f0cea47"). InnerVolumeSpecName "kube-api-access-hfjgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 02:47:37.286438 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:37.286353 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15c6bb65-87c0-4413-b15f-90938f0cea47-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "15c6bb65-87c0-4413-b15f-90938f0cea47" (UID: "15c6bb65-87c0-4413-b15f-90938f0cea47"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 02:47:37.384951 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:37.384920 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15c6bb65-87c0-4413-b15f-90938f0cea47-maas-api-tls\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:47:37.384951 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:37.384949 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfjgh\" (UniqueName: \"kubernetes.io/projected/15c6bb65-87c0-4413-b15f-90938f0cea47-kube-api-access-hfjgh\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:47:37.385119 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:37.384959 2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15c6bb65-87c0-4413-b15f-90938f0cea47-openshift-service-ca\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:47:37.888889 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:37.888851 2566 generic.go:356] "Generic (PLEG): container finished" podID="15c6bb65-87c0-4413-b15f-90938f0cea47" containerID="f17e5a335c718fdc6787877ac42452f8ee25ae399ab85bf3991e5f05bb37757f" exitCode=0 Jul 10 02:47:37.889060 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:37.888893 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bbcae0-bb867d558-z4xcs" Jul 10 02:47:37.889060 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:37.888931 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bbcae0-bb867d558-z4xcs" event={"ID":"15c6bb65-87c0-4413-b15f-90938f0cea47","Type":"ContainerDied","Data":"f17e5a335c718fdc6787877ac42452f8ee25ae399ab85bf3991e5f05bb37757f"} Jul 10 02:47:37.889060 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:37.888979 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bbcae0-bb867d558-z4xcs" event={"ID":"15c6bb65-87c0-4413-b15f-90938f0cea47","Type":"ContainerDied","Data":"fdfc6dd6b080f6d8c7665fe3daab294793e4b5367d7ec6c72614f36de7bc8584"} Jul 10 02:47:37.889060 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:37.889002 2566 scope.go:117] "RemoveContainer" containerID="f17e5a335c718fdc6787877ac42452f8ee25ae399ab85bf3991e5f05bb37757f" Jul 10 02:47:37.898661 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:37.898643 2566 scope.go:117] "RemoveContainer" containerID="f17e5a335c718fdc6787877ac42452f8ee25ae399ab85bf3991e5f05bb37757f" Jul 10 02:47:37.898950 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:47:37.898931 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f17e5a335c718fdc6787877ac42452f8ee25ae399ab85bf3991e5f05bb37757f\": container with ID starting with f17e5a335c718fdc6787877ac42452f8ee25ae399ab85bf3991e5f05bb37757f not found: ID does not exist" containerID="f17e5a335c718fdc6787877ac42452f8ee25ae399ab85bf3991e5f05bb37757f" Jul 10 02:47:37.899010 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:37.898956 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f17e5a335c718fdc6787877ac42452f8ee25ae399ab85bf3991e5f05bb37757f"} err="failed to get container status \"f17e5a335c718fdc6787877ac42452f8ee25ae399ab85bf3991e5f05bb37757f\": rpc error: code = NotFound desc = could not find container \"f17e5a335c718fdc6787877ac42452f8ee25ae399ab85bf3991e5f05bb37757f\": container with ID starting with f17e5a335c718fdc6787877ac42452f8ee25ae399ab85bf3991e5f05bb37757f not found: ID does not exist" Jul 10 02:47:37.912771 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:37.912747 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bbcae0-bb867d558-z4xcs"] Jul 10 02:47:37.917191 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:37.917173 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bbcae0-bb867d558-z4xcs"] Jul 10 02:47:39.017648 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:39.017614 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15c6bb65-87c0-4413-b15f-90938f0cea47" path="/var/lib/kubelet/pods/15c6bb65-87c0-4413-b15f-90938f0cea47/volumes" Jul 10 02:47:41.569022 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:41.568983 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:47:47.928524 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:47.928493 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:47:48.030055 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:48.030021 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:47:52.293180 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:52.293131 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:47:52.438495 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:52.438445 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:47:52.545083 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:52.545016 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:47:52.900191 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:52.900113 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:47:53.223906 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:53.223831 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:47:53.419205 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:53.419169 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:47:56.205029 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:56.204994 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:47:56.303604 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:56.303575 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:47:56.367397 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:56.367365 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dbaa7c-56fd7cd44f-v4zbr"] Jul 10 02:47:56.367754 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:56.367741 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15c6bb65-87c0-4413-b15f-90938f0cea47" containerName="maas-api" Jul 10 02:47:56.367803 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:56.367755 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="15c6bb65-87c0-4413-b15f-90938f0cea47" containerName="maas-api" Jul 10 02:47:56.367803 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:56.367769 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d97d4969-28ef-4159-a3a8-20d852a3255b" containerName="revoke-keys" Jul 10 02:47:56.367803 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:56.367774 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d97d4969-28ef-4159-a3a8-20d852a3255b" containerName="revoke-keys" Jul 10 02:47:56.367896 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:56.367823 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="15c6bb65-87c0-4413-b15f-90938f0cea47" containerName="maas-api" Jul 10 02:47:56.367896 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:56.367832 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d97d4969-28ef-4159-a3a8-20d852a3255b" containerName="revoke-keys" Jul 10 02:47:56.370850 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:56.370835 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dbaa7c-56fd7cd44f-v4zbr" Jul 10 02:47:56.373584 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:56.373564 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-dbaa7c-serving-cert\"" Jul 10 02:47:56.373703 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:56.373569 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 02:47:56.374735 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:56.374720 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 10 02:47:56.374790 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:56.374724 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-r7s9m\"" Jul 10 02:47:56.380000 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:56.379980 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dbaa7c-56fd7cd44f-v4zbr"] Jul 10 02:47:56.443803 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:56.443767 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00d4591f-450e-46bc-b7fe-0f625192c572-openshift-service-ca\") pod \"maas-api-e2e-api-a-dbaa7c-56fd7cd44f-v4zbr\" (UID: \"00d4591f-450e-46bc-b7fe-0f625192c572\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dbaa7c-56fd7cd44f-v4zbr" Jul 10 02:47:56.443803 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:56.443801 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzbmx\" (UniqueName: \"kubernetes.io/projected/00d4591f-450e-46bc-b7fe-0f625192c572-kube-api-access-lzbmx\") pod \"maas-api-e2e-api-a-dbaa7c-56fd7cd44f-v4zbr\" (UID: \"00d4591f-450e-46bc-b7fe-0f625192c572\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dbaa7c-56fd7cd44f-v4zbr" Jul 10 02:47:56.444018 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:56.443843 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00d4591f-450e-46bc-b7fe-0f625192c572-maas-api-tls\") pod \"maas-api-e2e-api-a-dbaa7c-56fd7cd44f-v4zbr\" (UID: \"00d4591f-450e-46bc-b7fe-0f625192c572\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dbaa7c-56fd7cd44f-v4zbr" Jul 10 02:47:56.475591 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:56.475518 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:47:56.545331 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:56.545294 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00d4591f-450e-46bc-b7fe-0f625192c572-openshift-service-ca\") pod \"maas-api-e2e-api-a-dbaa7c-56fd7cd44f-v4zbr\" (UID: \"00d4591f-450e-46bc-b7fe-0f625192c572\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dbaa7c-56fd7cd44f-v4zbr" Jul 10 02:47:56.545485 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:56.545339 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzbmx\" (UniqueName: \"kubernetes.io/projected/00d4591f-450e-46bc-b7fe-0f625192c572-kube-api-access-lzbmx\") pod \"maas-api-e2e-api-a-dbaa7c-56fd7cd44f-v4zbr\" (UID: \"00d4591f-450e-46bc-b7fe-0f625192c572\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dbaa7c-56fd7cd44f-v4zbr" Jul 10 02:47:56.545485 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:56.545404 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00d4591f-450e-46bc-b7fe-0f625192c572-maas-api-tls\") pod \"maas-api-e2e-api-a-dbaa7c-56fd7cd44f-v4zbr\" (UID: \"00d4591f-450e-46bc-b7fe-0f625192c572\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dbaa7c-56fd7cd44f-v4zbr" Jul 10 02:47:56.545564 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:47:56.545516 2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-dbaa7c-serving-cert: secret "maas-api-e2e-api-a-dbaa7c-serving-cert" not found Jul 10 02:47:56.545633 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:47:56.545622 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00d4591f-450e-46bc-b7fe-0f625192c572-maas-api-tls podName:00d4591f-450e-46bc-b7fe-0f625192c572 nodeName:}" failed. No retries permitted until 2026-07-10 02:47:57.045575895 +0000 UTC m=+2408.621852970 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/00d4591f-450e-46bc-b7fe-0f625192c572-maas-api-tls") pod "maas-api-e2e-api-a-dbaa7c-56fd7cd44f-v4zbr" (UID: "00d4591f-450e-46bc-b7fe-0f625192c572") : secret "maas-api-e2e-api-a-dbaa7c-serving-cert" not found Jul 10 02:47:56.545964 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:56.545946 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00d4591f-450e-46bc-b7fe-0f625192c572-openshift-service-ca\") pod \"maas-api-e2e-api-a-dbaa7c-56fd7cd44f-v4zbr\" (UID: \"00d4591f-450e-46bc-b7fe-0f625192c572\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dbaa7c-56fd7cd44f-v4zbr" Jul 10 02:47:56.559081 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:56.558848 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzbmx\" (UniqueName: \"kubernetes.io/projected/00d4591f-450e-46bc-b7fe-0f625192c572-kube-api-access-lzbmx\") pod \"maas-api-e2e-api-a-dbaa7c-56fd7cd44f-v4zbr\" (UID: \"00d4591f-450e-46bc-b7fe-0f625192c572\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dbaa7c-56fd7cd44f-v4zbr" Jul 10 02:47:56.976413 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:56.976379 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:47:57.048740 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:57.048710 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00d4591f-450e-46bc-b7fe-0f625192c572-maas-api-tls\") pod \"maas-api-e2e-api-a-dbaa7c-56fd7cd44f-v4zbr\" (UID: \"00d4591f-450e-46bc-b7fe-0f625192c572\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dbaa7c-56fd7cd44f-v4zbr" Jul 10 02:47:57.048909 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:47:57.048860 2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-dbaa7c-serving-cert: secret "maas-api-e2e-api-a-dbaa7c-serving-cert" not found Jul 10 02:47:57.048997 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:47:57.048929 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00d4591f-450e-46bc-b7fe-0f625192c572-maas-api-tls podName:00d4591f-450e-46bc-b7fe-0f625192c572 nodeName:}" failed. No retries permitted until 2026-07-10 02:47:58.048910262 +0000 UTC m=+2409.625187336 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/00d4591f-450e-46bc-b7fe-0f625192c572-maas-api-tls") pod "maas-api-e2e-api-a-dbaa7c-56fd7cd44f-v4zbr" (UID: "00d4591f-450e-46bc-b7fe-0f625192c572") : secret "maas-api-e2e-api-a-dbaa7c-serving-cert" not found Jul 10 02:47:57.075844 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:57.075820 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:47:57.178352 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:57.178316 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:47:58.057251 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:58.057214 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00d4591f-450e-46bc-b7fe-0f625192c572-maas-api-tls\") pod \"maas-api-e2e-api-a-dbaa7c-56fd7cd44f-v4zbr\" (UID: \"00d4591f-450e-46bc-b7fe-0f625192c572\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dbaa7c-56fd7cd44f-v4zbr" Jul 10 02:47:58.059527 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:58.059500 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00d4591f-450e-46bc-b7fe-0f625192c572-maas-api-tls\") pod \"maas-api-e2e-api-a-dbaa7c-56fd7cd44f-v4zbr\" (UID: \"00d4591f-450e-46bc-b7fe-0f625192c572\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dbaa7c-56fd7cd44f-v4zbr" Jul 10 02:47:58.181999 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:58.181966 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dbaa7c-56fd7cd44f-v4zbr" Jul 10 02:47:58.316184 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:58.316154 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dbaa7c-56fd7cd44f-v4zbr"] Jul 10 02:47:58.318162 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:47:58.318125 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00d4591f_450e_46bc_b7fe_0f625192c572.slice/crio-9f17fc78bbec2b0dd87fe4161eccded812e6f07e7ed80f3ea5a66b0b26fea6f2 WatchSource:0}: Error finding container 9f17fc78bbec2b0dd87fe4161eccded812e6f07e7ed80f3ea5a66b0b26fea6f2: Status 404 returned error can't find the container with id 9f17fc78bbec2b0dd87fe4161eccded812e6f07e7ed80f3ea5a66b0b26fea6f2 Jul 10 02:47:59.011857 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:59.011786 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dbaa7c-56fd7cd44f-v4zbr" event={"ID":"00d4591f-450e-46bc-b7fe-0f625192c572","Type":"ContainerStarted","Data":"a099d8a546bbc68ff03f5034aea32bb4440928474612aa3e9f9844ccd3dff246"} Jul 10 02:47:59.011857 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:59.011831 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dbaa7c-56fd7cd44f-v4zbr" event={"ID":"00d4591f-450e-46bc-b7fe-0f625192c572","Type":"ContainerStarted","Data":"9f17fc78bbec2b0dd87fe4161eccded812e6f07e7ed80f3ea5a66b0b26fea6f2"} Jul 10 02:47:59.028838 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:47:59.028785 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dbaa7c-56fd7cd44f-v4zbr" podStartSLOduration=2.61939398 podStartE2EDuration="3.028772052s" totalImagesPullingTime="409.378072ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:47:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:47:59.026141898 +0000 UTC m=+2410.602418983" watchObservedRunningTime="2026-07-10 02:47:59.028772052 +0000 UTC m=+2410.605049138" Jul 10 02:48:00.012770 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:00.012744 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dbaa7c-56fd7cd44f-v4zbr" Jul 10 02:48:00.741313 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:00.741278 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:01.224891 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:01.224854 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:01.568773 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:01.568726 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:01.868995 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:01.868891 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:04.292026 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:04.291976 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:04.394689 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:04.394655 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:04.405292 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:04.405261 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-20e0d1-6795cd4d86-svr6s"] Jul 10 02:48:04.409551 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:04.409523 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-20e0d1-6795cd4d86-svr6s" Jul 10 02:48:04.412205 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:04.412184 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-20e0d1-serving-cert\"" Jul 10 02:48:04.421387 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:04.421362 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-20e0d1-6795cd4d86-svr6s"] Jul 10 02:48:04.495475 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:04.495430 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:04.518692 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:04.518662 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ca4f462-9017-488a-afd3-cbc390cec512-openshift-service-ca\") pod \"maas-api-e2e-api-b-20e0d1-6795cd4d86-svr6s\" (UID: \"1ca4f462-9017-488a-afd3-cbc390cec512\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-20e0d1-6795cd4d86-svr6s" Jul 10 02:48:04.518846 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:04.518736 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ca4f462-9017-488a-afd3-cbc390cec512-maas-api-tls\") pod \"maas-api-e2e-api-b-20e0d1-6795cd4d86-svr6s\" (UID: \"1ca4f462-9017-488a-afd3-cbc390cec512\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-20e0d1-6795cd4d86-svr6s" Jul 10 02:48:04.518846 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:04.518778 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cncw6\" (UniqueName: \"kubernetes.io/projected/1ca4f462-9017-488a-afd3-cbc390cec512-kube-api-access-cncw6\") pod \"maas-api-e2e-api-b-20e0d1-6795cd4d86-svr6s\" (UID: \"1ca4f462-9017-488a-afd3-cbc390cec512\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-20e0d1-6795cd4d86-svr6s" Jul 10 02:48:04.612763 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:04.612682 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:04.619866 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:04.619844 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cncw6\" (UniqueName: \"kubernetes.io/projected/1ca4f462-9017-488a-afd3-cbc390cec512-kube-api-access-cncw6\") pod \"maas-api-e2e-api-b-20e0d1-6795cd4d86-svr6s\" (UID: \"1ca4f462-9017-488a-afd3-cbc390cec512\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-20e0d1-6795cd4d86-svr6s" Jul 10 02:48:04.619948 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:04.619886 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ca4f462-9017-488a-afd3-cbc390cec512-openshift-service-ca\") pod \"maas-api-e2e-api-b-20e0d1-6795cd4d86-svr6s\" (UID: \"1ca4f462-9017-488a-afd3-cbc390cec512\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-20e0d1-6795cd4d86-svr6s" Jul 10 02:48:04.620009 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:04.619946 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ca4f462-9017-488a-afd3-cbc390cec512-maas-api-tls\") pod \"maas-api-e2e-api-b-20e0d1-6795cd4d86-svr6s\" (UID: \"1ca4f462-9017-488a-afd3-cbc390cec512\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-20e0d1-6795cd4d86-svr6s" Jul 10 02:48:04.620518 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:04.620491 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ca4f462-9017-488a-afd3-cbc390cec512-openshift-service-ca\") pod \"maas-api-e2e-api-b-20e0d1-6795cd4d86-svr6s\" (UID: \"1ca4f462-9017-488a-afd3-cbc390cec512\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-20e0d1-6795cd4d86-svr6s" Jul 10 02:48:04.622288 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:04.622269 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ca4f462-9017-488a-afd3-cbc390cec512-maas-api-tls\") pod \"maas-api-e2e-api-b-20e0d1-6795cd4d86-svr6s\" (UID: \"1ca4f462-9017-488a-afd3-cbc390cec512\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-20e0d1-6795cd4d86-svr6s" Jul 10 02:48:04.629529 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:04.629506 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cncw6\" (UniqueName: \"kubernetes.io/projected/1ca4f462-9017-488a-afd3-cbc390cec512-kube-api-access-cncw6\") pod \"maas-api-e2e-api-b-20e0d1-6795cd4d86-svr6s\" (UID: \"1ca4f462-9017-488a-afd3-cbc390cec512\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-20e0d1-6795cd4d86-svr6s" Jul 10 02:48:04.724123 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:04.724099 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-20e0d1-6795cd4d86-svr6s" Jul 10 02:48:04.861226 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:04.861198 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-20e0d1-6795cd4d86-svr6s"] Jul 10 02:48:04.862853 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:48:04.862795 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ca4f462_9017_488a_afd3_cbc390cec512.slice/crio-d017cb92de870b197e2392dc01ceb7a43593b15b195910ab638c5a4be878d1b9 WatchSource:0}: Error finding container d017cb92de870b197e2392dc01ceb7a43593b15b195910ab638c5a4be878d1b9: Status 404 returned error can't find the container with id d017cb92de870b197e2392dc01ceb7a43593b15b195910ab638c5a4be878d1b9 Jul 10 02:48:05.041925 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:05.041890 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-20e0d1-6795cd4d86-svr6s" event={"ID":"1ca4f462-9017-488a-afd3-cbc390cec512","Type":"ContainerStarted","Data":"d017cb92de870b197e2392dc01ceb7a43593b15b195910ab638c5a4be878d1b9"} Jul 10 02:48:05.060710 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:05.060678 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:05.166870 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:05.166794 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:06.024225 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:06.024199 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dbaa7c-56fd7cd44f-v4zbr" Jul 10 02:48:06.050194 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:06.050161 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-20e0d1-6795cd4d86-svr6s" event={"ID":"1ca4f462-9017-488a-afd3-cbc390cec512","Type":"ContainerStarted","Data":"22cd0e4def10d9d8419dd967647bb083dd360556266a9178655ce85925e51c0a"} Jul 10 02:48:06.050448 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:06.050424 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-20e0d1-6795cd4d86-svr6s" Jul 10 02:48:06.068525 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:06.068475 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-20e0d1-6795cd4d86-svr6s" podStartSLOduration=1.680873182 podStartE2EDuration="2.068460332s" totalImagesPullingTime="387.58715ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:48:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:48:06.066559569 +0000 UTC m=+2417.642836655" watchObservedRunningTime="2026-07-10 02:48:06.068460332 +0000 UTC m=+2417.644737417" Jul 10 02:48:08.819633 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:08.819575 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:12.061344 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:12.061265 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-20e0d1-6795cd4d86-svr6s" Jul 10 02:48:17.019189 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:17.019153 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dbaa7c-xl6p9"] Jul 10 02:48:17.022932 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:17.022908 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dbaa7c-xl6p9" Jul 10 02:48:17.025454 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:17.025436 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rppvw\"" Jul 10 02:48:17.031859 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:17.031476 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dbaa7c-xl6p9"] Jul 10 02:48:17.139476 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:17.139448 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/920ae41f-7028-4e8b-9e8d-f18f8e666922-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dbaa7c-xl6p9\" (UID: \"920ae41f-7028-4e8b-9e8d-f18f8e666922\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dbaa7c-xl6p9" Jul 10 02:48:17.240417 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:17.240384 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/920ae41f-7028-4e8b-9e8d-f18f8e666922-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dbaa7c-xl6p9\" (UID: \"920ae41f-7028-4e8b-9e8d-f18f8e666922\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dbaa7c-xl6p9" Jul 10 02:48:17.241002 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:17.240983 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/920ae41f-7028-4e8b-9e8d-f18f8e666922-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dbaa7c-xl6p9\" (UID: \"920ae41f-7028-4e8b-9e8d-f18f8e666922\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dbaa7c-xl6p9" Jul 10 02:48:17.335318 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:17.335294 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dbaa7c-xl6p9" Jul 10 02:48:17.465211 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:17.465182 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dbaa7c-xl6p9"] Jul 10 02:48:17.467178 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:48:17.467151 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod920ae41f_7028_4e8b_9e8d_f18f8e666922.slice/crio-fc3fe3b18c174e7b5f742cc5f9da08652094190fcae1623328268ebfa8c2a2b7 WatchSource:0}: Error finding container fc3fe3b18c174e7b5f742cc5f9da08652094190fcae1623328268ebfa8c2a2b7: Status 404 returned error can't find the container with id fc3fe3b18c174e7b5f742cc5f9da08652094190fcae1623328268ebfa8c2a2b7 Jul 10 02:48:17.690528 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:17.690455 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:17.791743 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:17.791712 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:18.117212 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:18.117178 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dbaa7c-xl6p9" event={"ID":"920ae41f-7028-4e8b-9e8d-f18f8e666922","Type":"ContainerStarted","Data":"a1b52ccf29d96af8959533c4d60a008fd44cd11e4c35c449b073d2af0d5d0699"} Jul 10 02:48:18.117626 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:18.117230 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dbaa7c-xl6p9" event={"ID":"920ae41f-7028-4e8b-9e8d-f18f8e666922","Type":"ContainerStarted","Data":"fc3fe3b18c174e7b5f742cc5f9da08652094190fcae1623328268ebfa8c2a2b7"} Jul 10 02:48:18.134913 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:18.134862 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dbaa7c-xl6p9" podStartSLOduration=1.134846928 podStartE2EDuration="1.134846928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:48:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:48:18.13363293 +0000 UTC m=+2429.709910016" watchObservedRunningTime="2026-07-10 02:48:18.134846928 +0000 UTC m=+2429.711124032" Jul 10 02:48:19.123984 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:19.123949 2566 generic.go:356] "Generic (PLEG): container finished" podID="920ae41f-7028-4e8b-9e8d-f18f8e666922" containerID="a1b52ccf29d96af8959533c4d60a008fd44cd11e4c35c449b073d2af0d5d0699" exitCode=0 Jul 10 02:48:19.124352 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:19.124034 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dbaa7c-xl6p9" event={"ID":"920ae41f-7028-4e8b-9e8d-f18f8e666922","Type":"ContainerDied","Data":"a1b52ccf29d96af8959533c4d60a008fd44cd11e4c35c449b073d2af0d5d0699"} Jul 10 02:48:20.257752 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:20.257727 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dbaa7c-xl6p9" Jul 10 02:48:20.365984 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:20.365958 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/920ae41f-7028-4e8b-9e8d-f18f8e666922-maas-api-service-ca\") pod \"920ae41f-7028-4e8b-9e8d-f18f8e666922\" (UID: \"920ae41f-7028-4e8b-9e8d-f18f8e666922\") " Jul 10 02:48:20.366279 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:20.366258 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/920ae41f-7028-4e8b-9e8d-f18f8e666922-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "920ae41f-7028-4e8b-9e8d-f18f8e666922" (UID: "920ae41f-7028-4e8b-9e8d-f18f8e666922"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 02:48:20.466782 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:20.466707 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/920ae41f-7028-4e8b-9e8d-f18f8e666922-maas-api-service-ca\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:48:21.135872 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:21.135844 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dbaa7c-xl6p9" Jul 10 02:48:21.135872 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:21.135864 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dbaa7c-xl6p9" event={"ID":"920ae41f-7028-4e8b-9e8d-f18f8e666922","Type":"ContainerDied","Data":"fc3fe3b18c174e7b5f742cc5f9da08652094190fcae1623328268ebfa8c2a2b7"} Jul 10 02:48:21.136098 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:21.135903 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc3fe3b18c174e7b5f742cc5f9da08652094190fcae1623328268ebfa8c2a2b7" Jul 10 02:48:25.054774 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:25.054739 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:26.300861 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:26.300828 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:26.405627 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:26.405569 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:27.120471 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:27.120442 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dbaa7c-56fd7cd44f-v4zbr"] Jul 10 02:48:27.120713 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:27.120687 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dbaa7c-56fd7cd44f-v4zbr" podUID="00d4591f-450e-46bc-b7fe-0f625192c572" containerName="maas-api" containerID="cri-o://a099d8a546bbc68ff03f5034aea32bb4440928474612aa3e9f9844ccd3dff246" gracePeriod=30 Jul 10 02:48:27.186971 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:27.186938 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:27.379139 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:27.379068 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dbaa7c-56fd7cd44f-v4zbr" Jul 10 02:48:27.429767 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:27.429716 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzbmx\" (UniqueName: \"kubernetes.io/projected/00d4591f-450e-46bc-b7fe-0f625192c572-kube-api-access-lzbmx\") pod \"00d4591f-450e-46bc-b7fe-0f625192c572\" (UID: \"00d4591f-450e-46bc-b7fe-0f625192c572\") " Jul 10 02:48:27.429929 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:27.429788 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00d4591f-450e-46bc-b7fe-0f625192c572-maas-api-tls\") pod \"00d4591f-450e-46bc-b7fe-0f625192c572\" (UID: \"00d4591f-450e-46bc-b7fe-0f625192c572\") " Jul 10 02:48:27.429929 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:27.429829 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00d4591f-450e-46bc-b7fe-0f625192c572-openshift-service-ca\") pod \"00d4591f-450e-46bc-b7fe-0f625192c572\" (UID: \"00d4591f-450e-46bc-b7fe-0f625192c572\") " Jul 10 02:48:27.430639 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:27.430585 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00d4591f-450e-46bc-b7fe-0f625192c572-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "00d4591f-450e-46bc-b7fe-0f625192c572" (UID: "00d4591f-450e-46bc-b7fe-0f625192c572"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 02:48:27.433153 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:27.433117 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00d4591f-450e-46bc-b7fe-0f625192c572-kube-api-access-lzbmx" (OuterVolumeSpecName: "kube-api-access-lzbmx") pod "00d4591f-450e-46bc-b7fe-0f625192c572" (UID: "00d4591f-450e-46bc-b7fe-0f625192c572"). InnerVolumeSpecName "kube-api-access-lzbmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 02:48:27.433359 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:27.433340 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00d4591f-450e-46bc-b7fe-0f625192c572-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "00d4591f-450e-46bc-b7fe-0f625192c572" (UID: "00d4591f-450e-46bc-b7fe-0f625192c572"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 02:48:27.531575 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:27.531547 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzbmx\" (UniqueName: \"kubernetes.io/projected/00d4591f-450e-46bc-b7fe-0f625192c572-kube-api-access-lzbmx\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:48:27.531575 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:27.531573 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00d4591f-450e-46bc-b7fe-0f625192c572-maas-api-tls\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:48:27.531760 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:27.531587 2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00d4591f-450e-46bc-b7fe-0f625192c572-openshift-service-ca\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:48:28.175833 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:28.175800 2566 generic.go:356] "Generic (PLEG): container finished" podID="00d4591f-450e-46bc-b7fe-0f625192c572" containerID="a099d8a546bbc68ff03f5034aea32bb4440928474612aa3e9f9844ccd3dff246" exitCode=0 Jul 10 02:48:28.176016 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:28.175860 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dbaa7c-56fd7cd44f-v4zbr" Jul 10 02:48:28.176016 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:28.175891 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dbaa7c-56fd7cd44f-v4zbr" event={"ID":"00d4591f-450e-46bc-b7fe-0f625192c572","Type":"ContainerDied","Data":"a099d8a546bbc68ff03f5034aea32bb4440928474612aa3e9f9844ccd3dff246"} Jul 10 02:48:28.176016 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:28.175953 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dbaa7c-56fd7cd44f-v4zbr" event={"ID":"00d4591f-450e-46bc-b7fe-0f625192c572","Type":"ContainerDied","Data":"9f17fc78bbec2b0dd87fe4161eccded812e6f07e7ed80f3ea5a66b0b26fea6f2"} Jul 10 02:48:28.176016 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:28.175982 2566 scope.go:117] "RemoveContainer" containerID="a099d8a546bbc68ff03f5034aea32bb4440928474612aa3e9f9844ccd3dff246" Jul 10 02:48:28.186316 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:28.186294 2566 scope.go:117] "RemoveContainer" containerID="a099d8a546bbc68ff03f5034aea32bb4440928474612aa3e9f9844ccd3dff246" Jul 10 02:48:28.186563 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:48:28.186541 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a099d8a546bbc68ff03f5034aea32bb4440928474612aa3e9f9844ccd3dff246\": container with ID starting with a099d8a546bbc68ff03f5034aea32bb4440928474612aa3e9f9844ccd3dff246 not found: ID does not exist" containerID="a099d8a546bbc68ff03f5034aea32bb4440928474612aa3e9f9844ccd3dff246" Jul 10 02:48:28.186680 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:28.186569 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a099d8a546bbc68ff03f5034aea32bb4440928474612aa3e9f9844ccd3dff246"} err="failed to get container status \"a099d8a546bbc68ff03f5034aea32bb4440928474612aa3e9f9844ccd3dff246\": rpc error: code = NotFound desc = could not find container \"a099d8a546bbc68ff03f5034aea32bb4440928474612aa3e9f9844ccd3dff246\": container with ID starting with a099d8a546bbc68ff03f5034aea32bb4440928474612aa3e9f9844ccd3dff246 not found: ID does not exist" Jul 10 02:48:28.200208 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:28.200183 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dbaa7c-56fd7cd44f-v4zbr"] Jul 10 02:48:28.204544 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:28.204523 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dbaa7c-56fd7cd44f-v4zbr"] Jul 10 02:48:29.011510 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:29.011478 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00d4591f-450e-46bc-b7fe-0f625192c572" path="/var/lib/kubelet/pods/00d4591f-450e-46bc-b7fe-0f625192c572/volumes" Jul 10 02:48:32.128929 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:32.128896 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:38.402708 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:38.402671 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:38.501068 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:38.501040 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:42.876913 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:42.876881 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:43.117509 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:43.117478 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:43.242734 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:43.242649 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-20e0d1-l7jzr"] Jul 10 02:48:43.243090 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:43.243072 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00d4591f-450e-46bc-b7fe-0f625192c572" containerName="maas-api" Jul 10 02:48:43.243090 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:43.243090 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="00d4591f-450e-46bc-b7fe-0f625192c572" containerName="maas-api" Jul 10 02:48:43.243243 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:43.243105 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="920ae41f-7028-4e8b-9e8d-f18f8e666922" containerName="revoke-keys" Jul 10 02:48:43.243243 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:43.243112 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="920ae41f-7028-4e8b-9e8d-f18f8e666922" containerName="revoke-keys" Jul 10 02:48:43.243243 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:43.243173 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="00d4591f-450e-46bc-b7fe-0f625192c572" containerName="maas-api" Jul 10 02:48:43.243243 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:43.243182 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="920ae41f-7028-4e8b-9e8d-f18f8e666922" containerName="revoke-keys" Jul 10 02:48:43.247564 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:43.247547 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-20e0d1-l7jzr" Jul 10 02:48:43.250247 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:43.250229 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rppvw\"" Jul 10 02:48:43.255845 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:43.255632 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-20e0d1-l7jzr"] Jul 10 02:48:43.375978 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:43.375930 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/777e695a-9529-4542-8881-a45d000e10a6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-20e0d1-l7jzr\" (UID: \"777e695a-9529-4542-8881-a45d000e10a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-20e0d1-l7jzr" Jul 10 02:48:43.477261 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:43.477231 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/777e695a-9529-4542-8881-a45d000e10a6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-20e0d1-l7jzr\" (UID: \"777e695a-9529-4542-8881-a45d000e10a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-20e0d1-l7jzr" Jul 10 02:48:43.477833 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:43.477815 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/777e695a-9529-4542-8881-a45d000e10a6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-20e0d1-l7jzr\" (UID: \"777e695a-9529-4542-8881-a45d000e10a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-20e0d1-l7jzr" Jul 10 02:48:43.559614 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:43.559577 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-20e0d1-l7jzr" Jul 10 02:48:43.692454 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:43.692427 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-20e0d1-l7jzr"] Jul 10 02:48:43.694840 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:48:43.694810 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod777e695a_9529_4542_8881_a45d000e10a6.slice/crio-33c60365a85675ea7f4025a18226818fa8f05f771954a1af659bfd73a2645659 WatchSource:0}: Error finding container 33c60365a85675ea7f4025a18226818fa8f05f771954a1af659bfd73a2645659: Status 404 returned error can't find the container with id 33c60365a85675ea7f4025a18226818fa8f05f771954a1af659bfd73a2645659 Jul 10 02:48:44.269542 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:44.269435 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-20e0d1-l7jzr" event={"ID":"777e695a-9529-4542-8881-a45d000e10a6","Type":"ContainerStarted","Data":"2c6e7c9a9f57c182306d9f3f2728d51f09514c070fb14c68eca71736e77fabf7"} Jul 10 02:48:44.269542 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:44.269493 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-20e0d1-l7jzr" event={"ID":"777e695a-9529-4542-8881-a45d000e10a6","Type":"ContainerStarted","Data":"33c60365a85675ea7f4025a18226818fa8f05f771954a1af659bfd73a2645659"} Jul 10 02:48:44.286860 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:44.286802 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-20e0d1-l7jzr" podStartSLOduration=1.286782718 podStartE2EDuration="1.286782718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:48:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:48:44.284971732 +0000 UTC m=+2455.861248816" watchObservedRunningTime="2026-07-10 02:48:44.286782718 +0000 UTC m=+2455.863059805" Jul 10 02:48:45.277307 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:45.277272 2566 generic.go:356] "Generic (PLEG): container finished" podID="777e695a-9529-4542-8881-a45d000e10a6" containerID="2c6e7c9a9f57c182306d9f3f2728d51f09514c070fb14c68eca71736e77fabf7" exitCode=0 Jul 10 02:48:45.277676 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:45.277355 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-20e0d1-l7jzr" event={"ID":"777e695a-9529-4542-8881-a45d000e10a6","Type":"ContainerDied","Data":"2c6e7c9a9f57c182306d9f3f2728d51f09514c070fb14c68eca71736e77fabf7"} Jul 10 02:48:46.419742 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:46.419721 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-20e0d1-l7jzr" Jul 10 02:48:46.503880 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:46.503849 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/777e695a-9529-4542-8881-a45d000e10a6-maas-api-service-ca\") pod \"777e695a-9529-4542-8881-a45d000e10a6\" (UID: \"777e695a-9529-4542-8881-a45d000e10a6\") " Jul 10 02:48:46.504213 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:46.504190 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/777e695a-9529-4542-8881-a45d000e10a6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "777e695a-9529-4542-8881-a45d000e10a6" (UID: "777e695a-9529-4542-8881-a45d000e10a6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 02:48:46.605417 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:46.605332 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/777e695a-9529-4542-8881-a45d000e10a6-maas-api-service-ca\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:48:47.290056 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:47.290024 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-20e0d1-l7jzr" Jul 10 02:48:47.290239 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:47.290031 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-20e0d1-l7jzr" event={"ID":"777e695a-9529-4542-8881-a45d000e10a6","Type":"ContainerDied","Data":"33c60365a85675ea7f4025a18226818fa8f05f771954a1af659bfd73a2645659"} Jul 10 02:48:47.290239 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:47.290148 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33c60365a85675ea7f4025a18226818fa8f05f771954a1af659bfd73a2645659" Jul 10 02:48:53.776554 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:53.776518 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-20e0d1-6795cd4d86-svr6s"] Jul 10 02:48:53.776985 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:53.776837 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-20e0d1-6795cd4d86-svr6s" podUID="1ca4f462-9017-488a-afd3-cbc390cec512" containerName="maas-api" containerID="cri-o://22cd0e4def10d9d8419dd967647bb083dd360556266a9178655ce85925e51c0a" gracePeriod=30 Jul 10 02:48:53.801553 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:53.801528 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:54.027650 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:54.027576 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-20e0d1-6795cd4d86-svr6s" Jul 10 02:48:54.174674 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:54.174636 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cncw6\" (UniqueName: \"kubernetes.io/projected/1ca4f462-9017-488a-afd3-cbc390cec512-kube-api-access-cncw6\") pod \"1ca4f462-9017-488a-afd3-cbc390cec512\" (UID: \"1ca4f462-9017-488a-afd3-cbc390cec512\") " Jul 10 02:48:54.174862 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:54.174714 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ca4f462-9017-488a-afd3-cbc390cec512-maas-api-tls\") pod \"1ca4f462-9017-488a-afd3-cbc390cec512\" (UID: \"1ca4f462-9017-488a-afd3-cbc390cec512\") " Jul 10 02:48:54.174862 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:54.174744 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ca4f462-9017-488a-afd3-cbc390cec512-openshift-service-ca\") pod \"1ca4f462-9017-488a-afd3-cbc390cec512\" (UID: \"1ca4f462-9017-488a-afd3-cbc390cec512\") " Jul 10 02:48:54.175211 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:54.175180 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ca4f462-9017-488a-afd3-cbc390cec512-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1ca4f462-9017-488a-afd3-cbc390cec512" (UID: "1ca4f462-9017-488a-afd3-cbc390cec512"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 02:48:54.176775 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:54.176745 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ca4f462-9017-488a-afd3-cbc390cec512-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1ca4f462-9017-488a-afd3-cbc390cec512" (UID: "1ca4f462-9017-488a-afd3-cbc390cec512"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 02:48:54.176872 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:54.176786 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ca4f462-9017-488a-afd3-cbc390cec512-kube-api-access-cncw6" (OuterVolumeSpecName: "kube-api-access-cncw6") pod "1ca4f462-9017-488a-afd3-cbc390cec512" (UID: "1ca4f462-9017-488a-afd3-cbc390cec512"). InnerVolumeSpecName "kube-api-access-cncw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 02:48:54.275795 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:54.275770 2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ca4f462-9017-488a-afd3-cbc390cec512-openshift-service-ca\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:48:54.275795 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:54.275792 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cncw6\" (UniqueName: \"kubernetes.io/projected/1ca4f462-9017-488a-afd3-cbc390cec512-kube-api-access-cncw6\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:48:54.275962 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:54.275801 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ca4f462-9017-488a-afd3-cbc390cec512-maas-api-tls\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:48:54.331736 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:54.331681 2566 generic.go:356] "Generic (PLEG): container finished" podID="1ca4f462-9017-488a-afd3-cbc390cec512" containerID="22cd0e4def10d9d8419dd967647bb083dd360556266a9178655ce85925e51c0a" exitCode=0 Jul 10 02:48:54.331736 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:54.331723 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-20e0d1-6795cd4d86-svr6s" event={"ID":"1ca4f462-9017-488a-afd3-cbc390cec512","Type":"ContainerDied","Data":"22cd0e4def10d9d8419dd967647bb083dd360556266a9178655ce85925e51c0a"} Jul 10 02:48:54.331855 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:54.331777 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-20e0d1-6795cd4d86-svr6s" event={"ID":"1ca4f462-9017-488a-afd3-cbc390cec512","Type":"ContainerDied","Data":"d017cb92de870b197e2392dc01ceb7a43593b15b195910ab638c5a4be878d1b9"} Jul 10 02:48:54.331855 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:54.331798 2566 scope.go:117] "RemoveContainer" containerID="22cd0e4def10d9d8419dd967647bb083dd360556266a9178655ce85925e51c0a" Jul 10 02:48:54.331855 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:54.331730 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-20e0d1-6795cd4d86-svr6s" Jul 10 02:48:54.341480 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:54.341461 2566 scope.go:117] "RemoveContainer" containerID="22cd0e4def10d9d8419dd967647bb083dd360556266a9178655ce85925e51c0a" Jul 10 02:48:54.341759 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:48:54.341740 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22cd0e4def10d9d8419dd967647bb083dd360556266a9178655ce85925e51c0a\": container with ID starting with 22cd0e4def10d9d8419dd967647bb083dd360556266a9178655ce85925e51c0a not found: ID does not exist" containerID="22cd0e4def10d9d8419dd967647bb083dd360556266a9178655ce85925e51c0a" Jul 10 02:48:54.341831 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:54.341770 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22cd0e4def10d9d8419dd967647bb083dd360556266a9178655ce85925e51c0a"} err="failed to get container status \"22cd0e4def10d9d8419dd967647bb083dd360556266a9178655ce85925e51c0a\": rpc error: code = NotFound desc = could not find container \"22cd0e4def10d9d8419dd967647bb083dd360556266a9178655ce85925e51c0a\": container with ID starting with 22cd0e4def10d9d8419dd967647bb083dd360556266a9178655ce85925e51c0a not found: ID does not exist" Jul 10 02:48:54.357116 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:54.357090 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-20e0d1-6795cd4d86-svr6s"] Jul 10 02:48:54.367858 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:54.367834 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-20e0d1-6795cd4d86-svr6s"] Jul 10 02:48:55.011560 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:55.011530 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ca4f462-9017-488a-afd3-cbc390cec512" path="/var/lib/kubelet/pods/1ca4f462-9017-488a-afd3-cbc390cec512/volumes" Jul 10 02:48:58.345728 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:48:58.345687 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:49:04.330264 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:04.330224 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:49:04.430507 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:04.430472 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:49:09.091801 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:09.091767 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:49:09.348032 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:09.347921 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:49:09.703250 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:09.703169 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:49:10.052784 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:10.052749 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:49:10.875503 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:10.875454 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:49:12.258778 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:12.258742 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:49:12.355978 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:12.355946 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:49:12.643511 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:12.643464 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:49:12.734896 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:12.734865 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d5b98a-556f74c585-8g2kr"] Jul 10 02:49:12.735246 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:12.735223 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ca4f462-9017-488a-afd3-cbc390cec512" containerName="maas-api" Jul 10 02:49:12.735422 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:12.735404 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ca4f462-9017-488a-afd3-cbc390cec512" containerName="maas-api" Jul 10 02:49:12.735549 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:12.735461 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="777e695a-9529-4542-8881-a45d000e10a6" containerName="revoke-keys" Jul 10 02:49:12.735549 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:12.735471 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="777e695a-9529-4542-8881-a45d000e10a6" containerName="revoke-keys" Jul 10 02:49:12.735663 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:12.735612 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="777e695a-9529-4542-8881-a45d000e10a6" containerName="revoke-keys" Jul 10 02:49:12.735663 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:12.735627 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ca4f462-9017-488a-afd3-cbc390cec512" containerName="maas-api" Jul 10 02:49:12.739232 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:12.739214 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d5b98a-556f74c585-8g2kr" Jul 10 02:49:12.742203 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:12.742183 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-d5b98a-serving-cert\"" Jul 10 02:49:12.742312 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:12.742190 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 10 02:49:12.742525 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:12.742487 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:49:12.743547 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:12.743528 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 02:49:12.743547 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:12.743543 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-r7s9m\"" Jul 10 02:49:12.748911 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:12.748891 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d5b98a-556f74c585-8g2kr"] Jul 10 02:49:12.820160 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:12.820132 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:49:12.836263 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:12.836235 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57d0cdac-7351-415e-af39-8d70fd4957a5-maas-api-tls\") pod \"maas-api-e2e-shared-a-d5b98a-556f74c585-8g2kr\" (UID: \"57d0cdac-7351-415e-af39-8d70fd4957a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d5b98a-556f74c585-8g2kr" Jul 10 02:49:12.836410 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:12.836336 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdznw\" (UniqueName: \"kubernetes.io/projected/57d0cdac-7351-415e-af39-8d70fd4957a5-kube-api-access-mdznw\") pod \"maas-api-e2e-shared-a-d5b98a-556f74c585-8g2kr\" (UID: \"57d0cdac-7351-415e-af39-8d70fd4957a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d5b98a-556f74c585-8g2kr" Jul 10 02:49:12.836410 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:12.836378 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57d0cdac-7351-415e-af39-8d70fd4957a5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-d5b98a-556f74c585-8g2kr\" (UID: \"57d0cdac-7351-415e-af39-8d70fd4957a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d5b98a-556f74c585-8g2kr" Jul 10 02:49:12.937656 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:12.937566 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdznw\" (UniqueName: \"kubernetes.io/projected/57d0cdac-7351-415e-af39-8d70fd4957a5-kube-api-access-mdznw\") pod \"maas-api-e2e-shared-a-d5b98a-556f74c585-8g2kr\" (UID: \"57d0cdac-7351-415e-af39-8d70fd4957a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d5b98a-556f74c585-8g2kr" Jul 10 02:49:12.937656 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:12.937615 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57d0cdac-7351-415e-af39-8d70fd4957a5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-d5b98a-556f74c585-8g2kr\" (UID: \"57d0cdac-7351-415e-af39-8d70fd4957a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d5b98a-556f74c585-8g2kr" Jul 10 02:49:12.937900 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:12.937672 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57d0cdac-7351-415e-af39-8d70fd4957a5-maas-api-tls\") pod \"maas-api-e2e-shared-a-d5b98a-556f74c585-8g2kr\" (UID: \"57d0cdac-7351-415e-af39-8d70fd4957a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d5b98a-556f74c585-8g2kr" Jul 10 02:49:12.937900 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:49:12.937784 2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-d5b98a-serving-cert: secret "maas-api-e2e-shared-a-d5b98a-serving-cert" not found Jul 10 02:49:12.937900 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:49:12.937844 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57d0cdac-7351-415e-af39-8d70fd4957a5-maas-api-tls podName:57d0cdac-7351-415e-af39-8d70fd4957a5 nodeName:}" failed. No retries permitted until 2026-07-10 02:49:13.437827898 +0000 UTC m=+2485.014104978 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/57d0cdac-7351-415e-af39-8d70fd4957a5-maas-api-tls") pod "maas-api-e2e-shared-a-d5b98a-556f74c585-8g2kr" (UID: "57d0cdac-7351-415e-af39-8d70fd4957a5") : secret "maas-api-e2e-shared-a-d5b98a-serving-cert" not found Jul 10 02:49:12.938240 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:12.938221 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57d0cdac-7351-415e-af39-8d70fd4957a5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-d5b98a-556f74c585-8g2kr\" (UID: \"57d0cdac-7351-415e-af39-8d70fd4957a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d5b98a-556f74c585-8g2kr" Jul 10 02:49:12.950164 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:12.950142 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdznw\" (UniqueName: \"kubernetes.io/projected/57d0cdac-7351-415e-af39-8d70fd4957a5-kube-api-access-mdznw\") pod \"maas-api-e2e-shared-a-d5b98a-556f74c585-8g2kr\" (UID: \"57d0cdac-7351-415e-af39-8d70fd4957a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d5b98a-556f74c585-8g2kr" Jul 10 02:49:13.369650 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:13.369616 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:49:13.441531 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:13.441491 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57d0cdac-7351-415e-af39-8d70fd4957a5-maas-api-tls\") pod \"maas-api-e2e-shared-a-d5b98a-556f74c585-8g2kr\" (UID: \"57d0cdac-7351-415e-af39-8d70fd4957a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d5b98a-556f74c585-8g2kr" Jul 10 02:49:13.443963 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:13.443938 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57d0cdac-7351-415e-af39-8d70fd4957a5-maas-api-tls\") pod \"maas-api-e2e-shared-a-d5b98a-556f74c585-8g2kr\" (UID: \"57d0cdac-7351-415e-af39-8d70fd4957a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d5b98a-556f74c585-8g2kr" Jul 10 02:49:13.469385 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:13.469359 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:49:13.569959 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:13.569928 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:49:13.655861 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:13.655799 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d5b98a-556f74c585-8g2kr" Jul 10 02:49:13.788404 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:13.788382 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d5b98a-556f74c585-8g2kr"] Jul 10 02:49:13.790137 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:49:13.790089 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57d0cdac_7351_415e_af39_8d70fd4957a5.slice/crio-8be7a4270420516502278d4a8ea1373c32d5e0b3671b153cb218134f14d2a590 WatchSource:0}: Error finding container 8be7a4270420516502278d4a8ea1373c32d5e0b3671b153cb218134f14d2a590: Status 404 returned error can't find the container with id 8be7a4270420516502278d4a8ea1373c32d5e0b3671b153cb218134f14d2a590 Jul 10 02:49:14.450176 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:14.450132 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d5b98a-556f74c585-8g2kr" event={"ID":"57d0cdac-7351-415e-af39-8d70fd4957a5","Type":"ContainerStarted","Data":"2c639a5a7591eb0df7ca436acf0af85c1f8a6f261dd0b0f818a2e4b6cb04c308"} Jul 10 02:49:14.450568 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:14.450192 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d5b98a-556f74c585-8g2kr" event={"ID":"57d0cdac-7351-415e-af39-8d70fd4957a5","Type":"ContainerStarted","Data":"8be7a4270420516502278d4a8ea1373c32d5e0b3671b153cb218134f14d2a590"} Jul 10 02:49:14.450568 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:14.450221 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d5b98a-556f74c585-8g2kr" Jul 10 02:49:14.474864 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:14.474810 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d5b98a-556f74c585-8g2kr" podStartSLOduration=2.023828982 podStartE2EDuration="2.474791496s" totalImagesPullingTime="450.962514ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:49:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:49:14.471904396 +0000 UTC m=+2486.048181482" watchObservedRunningTime="2026-07-10 02:49:14.474791496 +0000 UTC m=+2486.051068581" Jul 10 02:49:17.187322 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:17.187283 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:49:17.545137 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:17.545106 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:49:17.890972 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:17.890879 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:49:17.997749 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:17.997720 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:49:20.461496 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:20.461465 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d5b98a-556f74c585-8g2kr" Jul 10 02:49:20.685552 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:20.685519 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:49:20.785467 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:20.785430 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:49:20.835877 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:20.835843 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-de8279-f76bdb884-l4rvl"] Jul 10 02:49:20.839736 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:20.839715 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de8279-f76bdb884-l4rvl" Jul 10 02:49:20.842202 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:20.842176 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-de8279-serving-cert\"" Jul 10 02:49:20.852689 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:20.852664 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-de8279-f76bdb884-l4rvl"] Jul 10 02:49:20.889896 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:20.889863 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:49:20.906787 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:20.906767 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61a83854-e226-4405-b282-12a1252da038-maas-api-tls\") pod \"maas-api-e2e-shared-b-de8279-f76bdb884-l4rvl\" (UID: \"61a83854-e226-4405-b282-12a1252da038\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de8279-f76bdb884-l4rvl" Jul 10 02:49:20.906871 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:20.906816 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr2pp\" (UniqueName: \"kubernetes.io/projected/61a83854-e226-4405-b282-12a1252da038-kube-api-access-gr2pp\") pod \"maas-api-e2e-shared-b-de8279-f76bdb884-l4rvl\" (UID: \"61a83854-e226-4405-b282-12a1252da038\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de8279-f76bdb884-l4rvl" Jul 10 02:49:20.906925 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:20.906883 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61a83854-e226-4405-b282-12a1252da038-openshift-service-ca\") pod \"maas-api-e2e-shared-b-de8279-f76bdb884-l4rvl\" (UID: \"61a83854-e226-4405-b282-12a1252da038\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de8279-f76bdb884-l4rvl" Jul 10 02:49:21.007541 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:21.007518 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61a83854-e226-4405-b282-12a1252da038-openshift-service-ca\") pod \"maas-api-e2e-shared-b-de8279-f76bdb884-l4rvl\" (UID: \"61a83854-e226-4405-b282-12a1252da038\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de8279-f76bdb884-l4rvl" Jul 10 02:49:21.007712 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:21.007568 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61a83854-e226-4405-b282-12a1252da038-maas-api-tls\") pod \"maas-api-e2e-shared-b-de8279-f76bdb884-l4rvl\" (UID: \"61a83854-e226-4405-b282-12a1252da038\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de8279-f76bdb884-l4rvl" Jul 10 02:49:21.007712 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:21.007627 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr2pp\" (UniqueName: \"kubernetes.io/projected/61a83854-e226-4405-b282-12a1252da038-kube-api-access-gr2pp\") pod \"maas-api-e2e-shared-b-de8279-f76bdb884-l4rvl\" (UID: \"61a83854-e226-4405-b282-12a1252da038\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de8279-f76bdb884-l4rvl" Jul 10 02:49:21.008156 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:21.008136 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61a83854-e226-4405-b282-12a1252da038-openshift-service-ca\") pod \"maas-api-e2e-shared-b-de8279-f76bdb884-l4rvl\" (UID: \"61a83854-e226-4405-b282-12a1252da038\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de8279-f76bdb884-l4rvl" Jul 10 02:49:21.010171 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:21.010148 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61a83854-e226-4405-b282-12a1252da038-maas-api-tls\") pod \"maas-api-e2e-shared-b-de8279-f76bdb884-l4rvl\" (UID: \"61a83854-e226-4405-b282-12a1252da038\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de8279-f76bdb884-l4rvl" Jul 10 02:49:21.025765 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:21.025741 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr2pp\" (UniqueName: \"kubernetes.io/projected/61a83854-e226-4405-b282-12a1252da038-kube-api-access-gr2pp\") pod \"maas-api-e2e-shared-b-de8279-f76bdb884-l4rvl\" (UID: \"61a83854-e226-4405-b282-12a1252da038\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de8279-f76bdb884-l4rvl" Jul 10 02:49:21.062078 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:21.062005 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:49:21.155326 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:21.155304 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de8279-f76bdb884-l4rvl" Jul 10 02:49:21.291863 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:21.291838 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-de8279-f76bdb884-l4rvl"] Jul 10 02:49:21.293541 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:49:21.293508 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61a83854_e226_4405_b282_12a1252da038.slice/crio-5fae698699e92006de323bf7c205acf463b146199e50c4cebc3caad6ec0aac14 WatchSource:0}: Error finding container 5fae698699e92006de323bf7c205acf463b146199e50c4cebc3caad6ec0aac14: Status 404 returned error can't find the container with id 5fae698699e92006de323bf7c205acf463b146199e50c4cebc3caad6ec0aac14 Jul 10 02:49:21.491363 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:21.491276 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de8279-f76bdb884-l4rvl" event={"ID":"61a83854-e226-4405-b282-12a1252da038","Type":"ContainerStarted","Data":"5fae698699e92006de323bf7c205acf463b146199e50c4cebc3caad6ec0aac14"} Jul 10 02:49:21.500403 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:21.500378 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:49:21.600722 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:21.600691 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:49:22.498410 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:22.498372 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de8279-f76bdb884-l4rvl" event={"ID":"61a83854-e226-4405-b282-12a1252da038","Type":"ContainerStarted","Data":"c64c90425ee801c64b65737deda684328c047f4b862eaedfedc2f09b2cfb3e0b"} Jul 10 02:49:22.498817 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:22.498437 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de8279-f76bdb884-l4rvl" Jul 10 02:49:22.519171 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:22.519129 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de8279-f76bdb884-l4rvl" podStartSLOduration=2.13924864 podStartE2EDuration="2.519114319s" totalImagesPullingTime="379.865679ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:49:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:49:22.515992483 +0000 UTC m=+2494.092269569" watchObservedRunningTime="2026-07-10 02:49:22.519114319 +0000 UTC m=+2494.095391462" Jul 10 02:49:25.195371 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:25.195340 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:49:28.510098 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:28.510069 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de8279-f76bdb884-l4rvl" Jul 10 02:49:35.715566 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:35.715531 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:49:35.815613 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:35.815567 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:49:40.079849 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:40.079768 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:49:43.644468 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:43.644434 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:49:43.746292 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:49:43.746267 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:50:00.788763 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:50:00.788733 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:50:05.330537 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:50:05.330501 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:50:46.545306 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:50:46.545272 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:50:46.645311 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:50:46.645280 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:50:47.568710 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:50:47.568679 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:50:48.680775 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:50:48.680745 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:52:12.683360 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:52:12.683322 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:52:13.666497 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:52:13.666463 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:52:43.286586 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:52:43.286500 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:53:13.581977 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:53:13.581942 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:53:54.993640 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:53:54.993584 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:53:58.930443 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:53:58.930403 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:53:59.882836 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:53:59.882793 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:54:13.247880 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:54:13.247797 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:54:13.347625 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:54:13.347565 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:54:15.399778 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:54:15.399747 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:54:17.803257 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:54:17.803225 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:54:36.665098 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:54:36.665042 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:54:36.779234 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:54:36.779205 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:54:44.642340 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:54:44.642303 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:54:44.746470 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:54:44.746433 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:55:08.481109 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:55:08.481076 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:55:56.968318 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:55:56.968237 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:55:58.090625 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:55:58.090562 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:56:13.873540 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:13.873509 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:56:17.194942 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:17.194903 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d5b98a-469cs"] Jul 10 02:56:17.198405 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:17.198389 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d5b98a-469cs" Jul 10 02:56:17.201141 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:17.201120 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rppvw\"" Jul 10 02:56:17.206402 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:17.206376 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d5b98a-469cs"] Jul 10 02:56:17.351496 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:17.351459 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05227166-5c0a-43f8-aea2-5d09dbe92d02-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-d5b98a-469cs\" (UID: \"05227166-5c0a-43f8-aea2-5d09dbe92d02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d5b98a-469cs" Jul 10 02:56:17.452629 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:17.452514 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05227166-5c0a-43f8-aea2-5d09dbe92d02-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-d5b98a-469cs\" (UID: \"05227166-5c0a-43f8-aea2-5d09dbe92d02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d5b98a-469cs" Jul 10 02:56:17.453204 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:17.453179 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05227166-5c0a-43f8-aea2-5d09dbe92d02-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-d5b98a-469cs\" (UID: \"05227166-5c0a-43f8-aea2-5d09dbe92d02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d5b98a-469cs" Jul 10 02:56:17.509284 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:17.509251 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d5b98a-469cs" Jul 10 02:56:17.646465 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:17.646434 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d5b98a-469cs"] Jul 10 02:56:17.648292 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:56:17.648257 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05227166_5c0a_43f8_aea2_5d09dbe92d02.slice/crio-bcbeae27497a726cb1cbe8b67f7a40ec4832be6ac233423209a04971dd995052 WatchSource:0}: Error finding container bcbeae27497a726cb1cbe8b67f7a40ec4832be6ac233423209a04971dd995052: Status 404 returned error can't find the container with id bcbeae27497a726cb1cbe8b67f7a40ec4832be6ac233423209a04971dd995052 Jul 10 02:56:17.844795 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:17.844758 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d5b98a-469cs" event={"ID":"05227166-5c0a-43f8-aea2-5d09dbe92d02","Type":"ContainerStarted","Data":"bcbeae27497a726cb1cbe8b67f7a40ec4832be6ac233423209a04971dd995052"} Jul 10 02:56:17.894382 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:17.894354 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:56:17.993396 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:17.993360 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:56:18.852245 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:18.852209 2566 generic.go:356] "Generic (PLEG): container finished" podID="05227166-5c0a-43f8-aea2-5d09dbe92d02" containerID="4777d079dc34139494593d0420a6223c5dbefbd458e314cf5bedcd44e9b73bff" exitCode=0 Jul 10 02:56:18.852718 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:18.852297 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d5b98a-469cs" event={"ID":"05227166-5c0a-43f8-aea2-5d09dbe92d02","Type":"ContainerDied","Data":"4777d079dc34139494593d0420a6223c5dbefbd458e314cf5bedcd44e9b73bff"} Jul 10 02:56:19.993499 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:19.993472 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d5b98a-469cs" Jul 10 02:56:20.180009 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:20.179937 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05227166-5c0a-43f8-aea2-5d09dbe92d02-maas-api-service-ca\") pod \"05227166-5c0a-43f8-aea2-5d09dbe92d02\" (UID: \"05227166-5c0a-43f8-aea2-5d09dbe92d02\") " Jul 10 02:56:20.180578 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:20.180277 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05227166-5c0a-43f8-aea2-5d09dbe92d02-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "05227166-5c0a-43f8-aea2-5d09dbe92d02" (UID: "05227166-5c0a-43f8-aea2-5d09dbe92d02"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 02:56:20.180578 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:20.180360 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05227166-5c0a-43f8-aea2-5d09dbe92d02-maas-api-service-ca\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:56:20.865088 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:20.865058 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d5b98a-469cs" Jul 10 02:56:20.865259 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:20.865087 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d5b98a-469cs" event={"ID":"05227166-5c0a-43f8-aea2-5d09dbe92d02","Type":"ContainerDied","Data":"bcbeae27497a726cb1cbe8b67f7a40ec4832be6ac233423209a04971dd995052"} Jul 10 02:56:20.865259 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:20.865130 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcbeae27497a726cb1cbe8b67f7a40ec4832be6ac233423209a04971dd995052" Jul 10 02:56:27.716202 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:27.716171 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d5b98a-556f74c585-8g2kr"] Jul 10 02:56:27.716738 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:27.716386 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d5b98a-556f74c585-8g2kr" podUID="57d0cdac-7351-415e-af39-8d70fd4957a5" containerName="maas-api" containerID="cri-o://2c639a5a7591eb0df7ca436acf0af85c1f8a6f261dd0b0f818a2e4b6cb04c308" gracePeriod=30 Jul 10 02:56:27.741754 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:27.741726 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:56:28.166439 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:28.166411 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d5b98a-556f74c585-8g2kr" Jul 10 02:56:28.235035 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:28.235006 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mdznw\" (UniqueName: \"kubernetes.io/projected/57d0cdac-7351-415e-af39-8d70fd4957a5-kube-api-access-mdznw\") pod \"57d0cdac-7351-415e-af39-8d70fd4957a5\" (UID: \"57d0cdac-7351-415e-af39-8d70fd4957a5\") " Jul 10 02:56:28.235185 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:28.235061 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57d0cdac-7351-415e-af39-8d70fd4957a5-maas-api-tls\") pod \"57d0cdac-7351-415e-af39-8d70fd4957a5\" (UID: \"57d0cdac-7351-415e-af39-8d70fd4957a5\") " Jul 10 02:56:28.235185 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:28.235089 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57d0cdac-7351-415e-af39-8d70fd4957a5-openshift-service-ca\") pod \"57d0cdac-7351-415e-af39-8d70fd4957a5\" (UID: \"57d0cdac-7351-415e-af39-8d70fd4957a5\") " Jul 10 02:56:28.235490 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:28.235460 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57d0cdac-7351-415e-af39-8d70fd4957a5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "57d0cdac-7351-415e-af39-8d70fd4957a5" (UID: "57d0cdac-7351-415e-af39-8d70fd4957a5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 02:56:28.237180 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:28.237160 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57d0cdac-7351-415e-af39-8d70fd4957a5-kube-api-access-mdznw" (OuterVolumeSpecName: "kube-api-access-mdznw") pod "57d0cdac-7351-415e-af39-8d70fd4957a5" (UID: "57d0cdac-7351-415e-af39-8d70fd4957a5"). InnerVolumeSpecName "kube-api-access-mdznw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 02:56:28.237270 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:28.237216 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57d0cdac-7351-415e-af39-8d70fd4957a5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "57d0cdac-7351-415e-af39-8d70fd4957a5" (UID: "57d0cdac-7351-415e-af39-8d70fd4957a5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 02:56:28.336197 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:28.336162 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdznw\" (UniqueName: \"kubernetes.io/projected/57d0cdac-7351-415e-af39-8d70fd4957a5-kube-api-access-mdznw\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:56:28.336197 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:28.336191 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57d0cdac-7351-415e-af39-8d70fd4957a5-maas-api-tls\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:56:28.336197 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:28.336203 2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57d0cdac-7351-415e-af39-8d70fd4957a5-openshift-service-ca\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:56:28.911402 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:28.911366 2566 generic.go:356] "Generic (PLEG): container finished" podID="57d0cdac-7351-415e-af39-8d70fd4957a5" containerID="2c639a5a7591eb0df7ca436acf0af85c1f8a6f261dd0b0f818a2e4b6cb04c308" exitCode=0 Jul 10 02:56:28.911922 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:28.911416 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d5b98a-556f74c585-8g2kr" Jul 10 02:56:28.911922 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:28.911420 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d5b98a-556f74c585-8g2kr" event={"ID":"57d0cdac-7351-415e-af39-8d70fd4957a5","Type":"ContainerDied","Data":"2c639a5a7591eb0df7ca436acf0af85c1f8a6f261dd0b0f818a2e4b6cb04c308"} Jul 10 02:56:28.911922 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:28.911475 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d5b98a-556f74c585-8g2kr" event={"ID":"57d0cdac-7351-415e-af39-8d70fd4957a5","Type":"ContainerDied","Data":"8be7a4270420516502278d4a8ea1373c32d5e0b3671b153cb218134f14d2a590"} Jul 10 02:56:28.911922 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:28.911509 2566 scope.go:117] "RemoveContainer" containerID="2c639a5a7591eb0df7ca436acf0af85c1f8a6f261dd0b0f818a2e4b6cb04c308" Jul 10 02:56:28.921306 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:28.921286 2566 scope.go:117] "RemoveContainer" containerID="2c639a5a7591eb0df7ca436acf0af85c1f8a6f261dd0b0f818a2e4b6cb04c308" Jul 10 02:56:28.921593 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:56:28.921574 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c639a5a7591eb0df7ca436acf0af85c1f8a6f261dd0b0f818a2e4b6cb04c308\": container with ID starting with 2c639a5a7591eb0df7ca436acf0af85c1f8a6f261dd0b0f818a2e4b6cb04c308 not found: ID does not exist" containerID="2c639a5a7591eb0df7ca436acf0af85c1f8a6f261dd0b0f818a2e4b6cb04c308" Jul 10 02:56:28.921670 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:28.921620 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c639a5a7591eb0df7ca436acf0af85c1f8a6f261dd0b0f818a2e4b6cb04c308"} err="failed to get container status \"2c639a5a7591eb0df7ca436acf0af85c1f8a6f261dd0b0f818a2e4b6cb04c308\": rpc error: code = NotFound desc = could not find container \"2c639a5a7591eb0df7ca436acf0af85c1f8a6f261dd0b0f818a2e4b6cb04c308\": container with ID starting with 2c639a5a7591eb0df7ca436acf0af85c1f8a6f261dd0b0f818a2e4b6cb04c308 not found: ID does not exist" Jul 10 02:56:28.935780 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:28.935726 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d5b98a-556f74c585-8g2kr"] Jul 10 02:56:28.938360 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:28.938348 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d5b98a-556f74c585-8g2kr"] Jul 10 02:56:29.012889 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:29.012860 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57d0cdac-7351-415e-af39-8d70fd4957a5" path="/var/lib/kubelet/pods/57d0cdac-7351-415e-af39-8d70fd4957a5/volumes" Jul 10 02:56:32.313098 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:32.313052 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:56:38.881797 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:38.881763 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:56:38.939193 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:38.939162 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:56:43.097107 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:43.097070 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:56:43.337329 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:43.337297 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:56:43.456374 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:43.456288 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-de8279-9dc2n"] Jul 10 02:56:43.456667 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:43.456654 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57d0cdac-7351-415e-af39-8d70fd4957a5" containerName="maas-api" Jul 10 02:56:43.456667 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:43.456668 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="57d0cdac-7351-415e-af39-8d70fd4957a5" containerName="maas-api" Jul 10 02:56:43.456788 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:43.456684 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05227166-5c0a-43f8-aea2-5d09dbe92d02" containerName="revoke-keys" Jul 10 02:56:43.456788 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:43.456690 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="05227166-5c0a-43f8-aea2-5d09dbe92d02" containerName="revoke-keys" Jul 10 02:56:43.456788 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:43.456757 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="05227166-5c0a-43f8-aea2-5d09dbe92d02" containerName="revoke-keys" Jul 10 02:56:43.456788 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:43.456768 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="57d0cdac-7351-415e-af39-8d70fd4957a5" containerName="maas-api" Jul 10 02:56:43.461142 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:43.461125 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-de8279-9dc2n" Jul 10 02:56:43.464165 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:43.464144 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rppvw\"" Jul 10 02:56:43.467821 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:43.467795 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-de8279-9dc2n"] Jul 10 02:56:43.549949 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:43.549918 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0b54617-441a-4ad0-8b2c-c67542a21397-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-de8279-9dc2n\" (UID: \"b0b54617-441a-4ad0-8b2c-c67542a21397\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-de8279-9dc2n" Jul 10 02:56:43.651373 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:43.651343 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0b54617-441a-4ad0-8b2c-c67542a21397-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-de8279-9dc2n\" (UID: \"b0b54617-441a-4ad0-8b2c-c67542a21397\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-de8279-9dc2n" Jul 10 02:56:43.651942 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:43.651923 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0b54617-441a-4ad0-8b2c-c67542a21397-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-de8279-9dc2n\" (UID: \"b0b54617-441a-4ad0-8b2c-c67542a21397\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-de8279-9dc2n" Jul 10 02:56:43.774581 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:43.774556 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-de8279-9dc2n" Jul 10 02:56:43.907064 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:43.907038 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-de8279-9dc2n"] Jul 10 02:56:43.908548 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:56:43.908526 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0b54617_441a_4ad0_8b2c_c67542a21397.slice/crio-7a1a5f293add432f75e78ab5bbac315cc1852827476b74c8e2b9db935a4f8d13 WatchSource:0}: Error finding container 7a1a5f293add432f75e78ab5bbac315cc1852827476b74c8e2b9db935a4f8d13: Status 404 returned error can't find the container with id 7a1a5f293add432f75e78ab5bbac315cc1852827476b74c8e2b9db935a4f8d13 Jul 10 02:56:44.006943 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:44.006912 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-de8279-9dc2n" event={"ID":"b0b54617-441a-4ad0-8b2c-c67542a21397","Type":"ContainerStarted","Data":"7a1a5f293add432f75e78ab5bbac315cc1852827476b74c8e2b9db935a4f8d13"} Jul 10 02:56:44.012019 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:44.011991 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:56:44.112668 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:44.112638 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:56:45.014897 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:45.014819 2566 generic.go:356] "Generic (PLEG): container finished" podID="b0b54617-441a-4ad0-8b2c-c67542a21397" containerID="7318c228457062b737f4eaae2821d1dd6abf31265e25506056ea79dc0c5fb09a" exitCode=0 Jul 10 02:56:45.015041 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:45.014897 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-de8279-9dc2n" event={"ID":"b0b54617-441a-4ad0-8b2c-c67542a21397","Type":"ContainerDied","Data":"7318c228457062b737f4eaae2821d1dd6abf31265e25506056ea79dc0c5fb09a"} Jul 10 02:56:46.154365 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:46.154342 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-de8279-9dc2n" Jul 10 02:56:46.274656 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:46.274564 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0b54617-441a-4ad0-8b2c-c67542a21397-maas-api-service-ca\") pod \"b0b54617-441a-4ad0-8b2c-c67542a21397\" (UID: \"b0b54617-441a-4ad0-8b2c-c67542a21397\") " Jul 10 02:56:46.274897 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:46.274876 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0b54617-441a-4ad0-8b2c-c67542a21397-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b0b54617-441a-4ad0-8b2c-c67542a21397" (UID: "b0b54617-441a-4ad0-8b2c-c67542a21397"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 02:56:46.375746 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:46.375717 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0b54617-441a-4ad0-8b2c-c67542a21397-maas-api-service-ca\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:56:47.027735 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:47.027712 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-de8279-9dc2n" Jul 10 02:56:47.027865 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:47.027755 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-de8279-9dc2n" event={"ID":"b0b54617-441a-4ad0-8b2c-c67542a21397","Type":"ContainerDied","Data":"7a1a5f293add432f75e78ab5bbac315cc1852827476b74c8e2b9db935a4f8d13"} Jul 10 02:56:47.027865 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:47.027797 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a1a5f293add432f75e78ab5bbac315cc1852827476b74c8e2b9db935a4f8d13" Jul 10 02:56:53.954373 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:53.954271 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-de8279-f76bdb884-l4rvl"] Jul 10 02:56:53.954876 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:53.954476 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de8279-f76bdb884-l4rvl" podUID="61a83854-e226-4405-b282-12a1252da038" containerName="maas-api" containerID="cri-o://c64c90425ee801c64b65737deda684328c047f4b862eaedfedc2f09b2cfb3e0b" gracePeriod=30 Jul 10 02:56:53.983991 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:53.983963 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:56:54.415524 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:54.415500 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de8279-f76bdb884-l4rvl" Jul 10 02:56:54.543802 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:54.543769 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61a83854-e226-4405-b282-12a1252da038-openshift-service-ca\") pod \"61a83854-e226-4405-b282-12a1252da038\" (UID: \"61a83854-e226-4405-b282-12a1252da038\") " Jul 10 02:56:54.543972 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:54.543833 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61a83854-e226-4405-b282-12a1252da038-maas-api-tls\") pod \"61a83854-e226-4405-b282-12a1252da038\" (UID: \"61a83854-e226-4405-b282-12a1252da038\") " Jul 10 02:56:54.543972 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:54.543851 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gr2pp\" (UniqueName: \"kubernetes.io/projected/61a83854-e226-4405-b282-12a1252da038-kube-api-access-gr2pp\") pod \"61a83854-e226-4405-b282-12a1252da038\" (UID: \"61a83854-e226-4405-b282-12a1252da038\") " Jul 10 02:56:54.544287 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:54.544252 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61a83854-e226-4405-b282-12a1252da038-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "61a83854-e226-4405-b282-12a1252da038" (UID: "61a83854-e226-4405-b282-12a1252da038"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 02:56:54.545967 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:54.545939 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61a83854-e226-4405-b282-12a1252da038-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "61a83854-e226-4405-b282-12a1252da038" (UID: "61a83854-e226-4405-b282-12a1252da038"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 02:56:54.546056 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:54.546038 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61a83854-e226-4405-b282-12a1252da038-kube-api-access-gr2pp" (OuterVolumeSpecName: "kube-api-access-gr2pp") pod "61a83854-e226-4405-b282-12a1252da038" (UID: "61a83854-e226-4405-b282-12a1252da038"). InnerVolumeSpecName "kube-api-access-gr2pp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 02:56:54.645422 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:54.645387 2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61a83854-e226-4405-b282-12a1252da038-openshift-service-ca\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:56:54.645422 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:54.645413 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61a83854-e226-4405-b282-12a1252da038-maas-api-tls\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:56:54.645422 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:54.645424 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr2pp\" (UniqueName: \"kubernetes.io/projected/61a83854-e226-4405-b282-12a1252da038-kube-api-access-gr2pp\") on node \"ip-10-0-130-65.ec2.internal\" DevicePath \"\"" Jul 10 02:56:55.080945 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:55.080914 2566 generic.go:356] "Generic (PLEG): container finished" podID="61a83854-e226-4405-b282-12a1252da038" containerID="c64c90425ee801c64b65737deda684328c047f4b862eaedfedc2f09b2cfb3e0b" exitCode=0 Jul 10 02:56:55.081371 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:55.080955 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de8279-f76bdb884-l4rvl" Jul 10 02:56:55.081371 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:55.081002 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de8279-f76bdb884-l4rvl" event={"ID":"61a83854-e226-4405-b282-12a1252da038","Type":"ContainerDied","Data":"c64c90425ee801c64b65737deda684328c047f4b862eaedfedc2f09b2cfb3e0b"} Jul 10 02:56:55.081371 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:55.081055 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-de8279-f76bdb884-l4rvl" event={"ID":"61a83854-e226-4405-b282-12a1252da038","Type":"ContainerDied","Data":"5fae698699e92006de323bf7c205acf463b146199e50c4cebc3caad6ec0aac14"} Jul 10 02:56:55.081371 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:55.081082 2566 scope.go:117] "RemoveContainer" containerID="c64c90425ee801c64b65737deda684328c047f4b862eaedfedc2f09b2cfb3e0b" Jul 10 02:56:55.090346 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:55.090327 2566 scope.go:117] "RemoveContainer" containerID="c64c90425ee801c64b65737deda684328c047f4b862eaedfedc2f09b2cfb3e0b" Jul 10 02:56:55.090614 ip-10-0-130-65 kubenswrapper[2566]: E0710 02:56:55.090572 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c64c90425ee801c64b65737deda684328c047f4b862eaedfedc2f09b2cfb3e0b\": container with ID starting with c64c90425ee801c64b65737deda684328c047f4b862eaedfedc2f09b2cfb3e0b not found: ID does not exist" containerID="c64c90425ee801c64b65737deda684328c047f4b862eaedfedc2f09b2cfb3e0b" Jul 10 02:56:55.090674 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:55.090615 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c64c90425ee801c64b65737deda684328c047f4b862eaedfedc2f09b2cfb3e0b"} err="failed to get container status \"c64c90425ee801c64b65737deda684328c047f4b862eaedfedc2f09b2cfb3e0b\": rpc error: code = NotFound desc = could not find container \"c64c90425ee801c64b65737deda684328c047f4b862eaedfedc2f09b2cfb3e0b\": container with ID starting with c64c90425ee801c64b65737deda684328c047f4b862eaedfedc2f09b2cfb3e0b not found: ID does not exist" Jul 10 02:56:55.101041 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:55.101018 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-de8279-f76bdb884-l4rvl"] Jul 10 02:56:55.105868 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:55.105842 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-de8279-f76bdb884-l4rvl"] Jul 10 02:56:57.011169 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:57.011140 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61a83854-e226-4405-b282-12a1252da038" path="/var/lib/kubelet/pods/61a83854-e226-4405-b282-12a1252da038/volumes" Jul 10 02:56:58.570545 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:56:58.570511 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:57:05.203965 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:05.203928 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:57:05.304052 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:05.304020 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:57:09.286923 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:09.286879 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:57:09.521789 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:09.521751 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:57:15.881812 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:15.881742 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-78b9cbb96-rl82t_d85d3688-f7b6-4739-a498-33484bd9b45f/copy-manifests/0.log" Jul 10 02:57:15.895694 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:15.895669 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-78b9cbb96-rl82t_d85d3688-f7b6-4739-a498-33484bd9b45f/manager/0.log" Jul 10 02:57:16.553722 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:16.553694 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29727510-sg6q6_aba74ed4-e7e7-4266-a503-fe797ef3b2ec/cleanup/0.log" Jul 10 02:57:16.675618 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:16.675573 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29727525-9gdn7_030df5fa-92a4-45fe-8e7f-32da38cad42f/cleanup/0.log" Jul 10 02:57:16.795817 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:16.795783 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-0a1b34cb-ct4fp_f8fbd472-02cc-43e1-97a2-0c0e3b64c734/revoke-keys/0.log" Jul 10 02:57:16.916065 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:16.915982 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-bc69ef18-r8mp9_d0350ca0-9993-4f12-ba84-93164b133483/revoke-keys/0.log" Jul 10 02:57:17.036787 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:17.036764 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-a-dbaa7c-xl6p9_920ae41f-7028-4e8b-9e8d-f18f8e666922/revoke-keys/0.log" Jul 10 02:57:17.154222 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:17.154197 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-20e0d1-l7jzr_777e695a-9529-4542-8881-a45d000e10a6/revoke-keys/0.log" Jul 10 02:57:17.276480 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:17.276458 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-derive-987cdfe4-75lj2_b99df84c-c677-4cb5-8b24-1149b5092624/revoke-keys/0.log" Jul 10 02:57:17.396558 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:17.396527 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-a-d76849-7m772_d88b769e-b5d5-4e09-8f37-ebdd1efdd639/revoke-keys/0.log" Jul 10 02:57:17.515214 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:17.515192 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-b-bbcae0-nc5mg_d97d4969-28ef-4159-a3a8-20d852a3255b/revoke-keys/0.log" Jul 10 02:57:17.635247 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:17.635182 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-abb8864d-ssmpn_ec2a7ed0-25ee-4f2b-9fce-c1d98e0205f0/revoke-keys/0.log" Jul 10 02:57:17.751241 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:17.751214 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-b3d83c99-4rprd_d8f49b08-a63d-4b40-931c-ed5d4d4e74fa/revoke-keys/0.log" Jul 10 02:57:17.872279 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:17.872251 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-a-d5b98a-469cs_05227166-5c0a-43f8-aea2-5d09dbe92d02/revoke-keys/0.log" Jul 10 02:57:17.987163 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:17.987098 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-b-de8279-9dc2n_b0b54617-441a-4ad0-8b2c-c67542a21397/revoke-keys/0.log" Jul 10 02:57:18.929344 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:18.929313 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594bstx_92d2a4bc-f048-4da4-a974-4787843d0874/util/0.log" Jul 10 02:57:18.936958 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:18.936934 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594bstx_92d2a4bc-f048-4da4-a974-4787843d0874/pull/0.log" Jul 10 02:57:18.944478 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:18.944461 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594bstx_92d2a4bc-f048-4da4-a974-4787843d0874/extract/0.log" Jul 10 02:57:19.064683 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:19.064661 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02rdkd_0545a39d-110a-43eb-9e80-9d1ca3643afd/util/0.log" Jul 10 02:57:19.072379 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:19.072360 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02rdkd_0545a39d-110a-43eb-9e80-9d1ca3643afd/pull/0.log" Jul 10 02:57:19.080203 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:19.080178 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02rdkd_0545a39d-110a-43eb-9e80-9d1ca3643afd/extract/0.log" Jul 10 02:57:19.201376 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:19.201310 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq9xc_44f94038-e230-4433-985a-91a3383be807/util/0.log" Jul 10 02:57:19.208802 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:19.208771 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq9xc_44f94038-e230-4433-985a-91a3383be807/pull/0.log" Jul 10 02:57:19.215990 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:19.215973 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq9xc_44f94038-e230-4433-985a-91a3383be807/extract/0.log" Jul 10 02:57:19.335399 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:19.335378 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cfkcm_f870d1d5-2091-49f5-b738-bcf5fa6bcfed/util/0.log" Jul 10 02:57:19.342359 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:19.342342 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cfkcm_f870d1d5-2091-49f5-b738-bcf5fa6bcfed/pull/0.log" Jul 10 02:57:19.349447 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:19.349432 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cfkcm_f870d1d5-2091-49f5-b738-bcf5fa6bcfed/extract/0.log" Jul 10 02:57:19.725151 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:19.725120 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-mnsp8_61031f36-a534-40fc-b736-35cb1712ffd3/manager/0.log" Jul 10 02:57:20.367047 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:20.367018 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-9wzjx_57b0005a-50f5-4a9e-ad28-e3651574a834/manager/0.log" Jul 10 02:57:20.864318 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:20.864289 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-9ts97_49d07846-4ea0-496e-843b-90a8920ea7bc/discovery/0.log" Jul 10 02:57:21.618008 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:21.617978 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6cd68c6448-czfxh_82ce150d-af95-4650-a996-fbe49696196f/router/0.log" Jul 10 02:57:21.985324 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:21.985260 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-l6zwm_fe1e3e8b-8a7d-4580-a413-5bbe6df30fe9/main/0.log" Jul 10 02:57:22.348302 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:22.348275 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbx5wbv_63dccc93-8fe7-443d-b84f-11d2c041c13b/main/0.log" Jul 10 02:57:27.237343 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:27.237312 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:57:27.336210 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:27.336180 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" limit=1024 Jul 10 02:57:30.068896 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:30.068861 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pcfzz_709370a6-295c-4330-bfa5-50ca602a28fb/global-pull-secret-syncer/0.log" Jul 10 02:57:30.159873 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:30.159845 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-cprsf_452d0117-6775-4e71-acc6-4e2fc639e369/konnectivity-agent/0.log" Jul 10 02:57:30.266301 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:30.266280 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-65.ec2.internal_305afbb58c749f5158dc47cb0c52154f/haproxy/0.log" Jul 10 02:57:34.389172 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:34.389146 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594bstx_92d2a4bc-f048-4da4-a974-4787843d0874/extract/0.log" Jul 10 02:57:34.413902 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:34.413878 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594bstx_92d2a4bc-f048-4da4-a974-4787843d0874/util/0.log" Jul 10 02:57:34.438470 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:34.438448 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594bstx_92d2a4bc-f048-4da4-a974-4787843d0874/pull/0.log" Jul 10 02:57:34.470997 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:34.470975 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02rdkd_0545a39d-110a-43eb-9e80-9d1ca3643afd/extract/0.log" Jul 10 02:57:34.496211 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:34.496183 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02rdkd_0545a39d-110a-43eb-9e80-9d1ca3643afd/util/0.log" Jul 10 02:57:34.528741 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:34.528717 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02rdkd_0545a39d-110a-43eb-9e80-9d1ca3643afd/pull/0.log" Jul 10 02:57:34.571336 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:34.571312 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq9xc_44f94038-e230-4433-985a-91a3383be807/extract/0.log" Jul 10 02:57:34.598114 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:34.598085 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq9xc_44f94038-e230-4433-985a-91a3383be807/util/0.log" Jul 10 02:57:34.630060 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:34.630038 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tq9xc_44f94038-e230-4433-985a-91a3383be807/pull/0.log" Jul 10 02:57:34.659987 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:34.659930 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cfkcm_f870d1d5-2091-49f5-b738-bcf5fa6bcfed/extract/0.log" Jul 10 02:57:34.685719 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:34.685702 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cfkcm_f870d1d5-2091-49f5-b738-bcf5fa6bcfed/util/0.log" Jul 10 02:57:34.710762 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:34.710739 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cfkcm_f870d1d5-2091-49f5-b738-bcf5fa6bcfed/pull/0.log" Jul 10 02:57:35.319829 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:35.319801 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-mnsp8_61031f36-a534-40fc-b736-35cb1712ffd3/manager/0.log" Jul 10 02:57:35.706034 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:35.705921 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-9wzjx_57b0005a-50f5-4a9e-ad28-e3651574a834/manager/0.log" Jul 10 02:57:37.667714 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:37.667686 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-nznmv_4642e40c-4306-467a-b96a-87311adc28e5/kube-state-metrics/0.log" Jul 10 02:57:37.690347 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:37.690326 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-nznmv_4642e40c-4306-467a-b96a-87311adc28e5/kube-rbac-proxy-main/0.log" Jul 10 02:57:37.715486 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:37.715467 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-nznmv_4642e40c-4306-467a-b96a-87311adc28e5/kube-rbac-proxy-self/0.log" Jul 10 02:57:37.747257 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:37.747240 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7bd9d9f484-mmhsv_7a42cfbb-60df-4742-80e4-c637ea5555fd/metrics-server/0.log" Jul 10 02:57:37.774031 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:37.774013 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-5vbk8_ac1608c3-32c1-4ccf-9c09-300a8524a4ae/monitoring-plugin/0.log" Jul 10 02:57:37.992514 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:37.992448 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t64f9_fa530c6a-9f5d-4919-bb59-f3e7752a6c79/node-exporter/0.log" Jul 10 02:57:38.017189 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:38.017168 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t64f9_fa530c6a-9f5d-4919-bb59-f3e7752a6c79/kube-rbac-proxy/0.log" Jul 10 02:57:38.041908 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:38.041888 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t64f9_fa530c6a-9f5d-4919-bb59-f3e7752a6c79/init-textfile/0.log" Jul 10 02:57:38.072015 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:38.071998 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-tplmw_ef076029-9853-4835-b6ac-ee0a295c23fc/kube-rbac-proxy-main/0.log" Jul 10 02:57:38.096217 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:38.096198 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-tplmw_ef076029-9853-4835-b6ac-ee0a295c23fc/kube-rbac-proxy-self/0.log" Jul 10 02:57:38.120893 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:38.120874 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-tplmw_ef076029-9853-4835-b6ac-ee0a295c23fc/openshift-state-metrics/0.log" Jul 10 02:57:39.640285 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:39.640252 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-s4bld/perf-node-gather-daemonset-blfc8"] Jul 10 02:57:39.640799 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:39.640780 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61a83854-e226-4405-b282-12a1252da038" containerName="maas-api" Jul 10 02:57:39.640875 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:39.640802 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="61a83854-e226-4405-b282-12a1252da038" containerName="maas-api" Jul 10 02:57:39.640875 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:39.640850 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0b54617-441a-4ad0-8b2c-c67542a21397" containerName="revoke-keys" Jul 10 02:57:39.640875 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:39.640860 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0b54617-441a-4ad0-8b2c-c67542a21397" containerName="revoke-keys" Jul 10 02:57:39.641023 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:39.640957 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="61a83854-e226-4405-b282-12a1252da038" containerName="maas-api" Jul 10 02:57:39.641023 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:39.640970 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0b54617-441a-4ad0-8b2c-c67542a21397" containerName="revoke-keys" Jul 10 02:57:39.644265 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:39.644244 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s4bld/perf-node-gather-daemonset-blfc8" Jul 10 02:57:39.647228 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:39.647205 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-s4bld\"/\"default-dockercfg-q2jnq\"" Jul 10 02:57:39.648116 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:39.648096 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s4bld\"/\"kube-root-ca.crt\"" Jul 10 02:57:39.648188 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:39.648162 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s4bld\"/\"openshift-service-ca.crt\"" Jul 10 02:57:39.653461 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:39.653440 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s4bld/perf-node-gather-daemonset-blfc8"] Jul 10 02:57:39.713946 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:39.713919 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7thf\" (UniqueName: \"kubernetes.io/projected/a25adb99-ccc9-4dfe-ab8a-e07a082fa0e3-kube-api-access-f7thf\") pod \"perf-node-gather-daemonset-blfc8\" (UID: \"a25adb99-ccc9-4dfe-ab8a-e07a082fa0e3\") " pod="openshift-must-gather-s4bld/perf-node-gather-daemonset-blfc8" Jul 10 02:57:39.714084 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:39.713984 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a25adb99-ccc9-4dfe-ab8a-e07a082fa0e3-proc\") pod \"perf-node-gather-daemonset-blfc8\" (UID: \"a25adb99-ccc9-4dfe-ab8a-e07a082fa0e3\") " pod="openshift-must-gather-s4bld/perf-node-gather-daemonset-blfc8" Jul 10 02:57:39.714084 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:39.714022 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a25adb99-ccc9-4dfe-ab8a-e07a082fa0e3-sys\") pod \"perf-node-gather-daemonset-blfc8\" (UID: \"a25adb99-ccc9-4dfe-ab8a-e07a082fa0e3\") " pod="openshift-must-gather-s4bld/perf-node-gather-daemonset-blfc8" Jul 10 02:57:39.714084 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:39.714044 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a25adb99-ccc9-4dfe-ab8a-e07a082fa0e3-podres\") pod \"perf-node-gather-daemonset-blfc8\" (UID: \"a25adb99-ccc9-4dfe-ab8a-e07a082fa0e3\") " pod="openshift-must-gather-s4bld/perf-node-gather-daemonset-blfc8" Jul 10 02:57:39.714205 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:39.714121 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a25adb99-ccc9-4dfe-ab8a-e07a082fa0e3-lib-modules\") pod \"perf-node-gather-daemonset-blfc8\" (UID: \"a25adb99-ccc9-4dfe-ab8a-e07a082fa0e3\") " pod="openshift-must-gather-s4bld/perf-node-gather-daemonset-blfc8" Jul 10 02:57:39.815332 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:39.815298 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a25adb99-ccc9-4dfe-ab8a-e07a082fa0e3-proc\") pod \"perf-node-gather-daemonset-blfc8\" (UID: \"a25adb99-ccc9-4dfe-ab8a-e07a082fa0e3\") " pod="openshift-must-gather-s4bld/perf-node-gather-daemonset-blfc8" Jul 10 02:57:39.815505 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:39.815350 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a25adb99-ccc9-4dfe-ab8a-e07a082fa0e3-sys\") pod \"perf-node-gather-daemonset-blfc8\" (UID: \"a25adb99-ccc9-4dfe-ab8a-e07a082fa0e3\") " pod="openshift-must-gather-s4bld/perf-node-gather-daemonset-blfc8" Jul 10 02:57:39.815505 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:39.815372 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a25adb99-ccc9-4dfe-ab8a-e07a082fa0e3-podres\") pod \"perf-node-gather-daemonset-blfc8\" (UID: \"a25adb99-ccc9-4dfe-ab8a-e07a082fa0e3\") " pod="openshift-must-gather-s4bld/perf-node-gather-daemonset-blfc8" Jul 10 02:57:39.815505 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:39.815398 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a25adb99-ccc9-4dfe-ab8a-e07a082fa0e3-lib-modules\") pod \"perf-node-gather-daemonset-blfc8\" (UID: \"a25adb99-ccc9-4dfe-ab8a-e07a082fa0e3\") " pod="openshift-must-gather-s4bld/perf-node-gather-daemonset-blfc8" Jul 10 02:57:39.815505 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:39.815417 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a25adb99-ccc9-4dfe-ab8a-e07a082fa0e3-proc\") pod \"perf-node-gather-daemonset-blfc8\" (UID: \"a25adb99-ccc9-4dfe-ab8a-e07a082fa0e3\") " pod="openshift-must-gather-s4bld/perf-node-gather-daemonset-blfc8" Jul 10 02:57:39.815505 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:39.815461 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a25adb99-ccc9-4dfe-ab8a-e07a082fa0e3-sys\") pod \"perf-node-gather-daemonset-blfc8\" (UID: \"a25adb99-ccc9-4dfe-ab8a-e07a082fa0e3\") " pod="openshift-must-gather-s4bld/perf-node-gather-daemonset-blfc8" Jul 10 02:57:39.815505 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:39.815501 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7thf\" (UniqueName: \"kubernetes.io/projected/a25adb99-ccc9-4dfe-ab8a-e07a082fa0e3-kube-api-access-f7thf\") pod \"perf-node-gather-daemonset-blfc8\" (UID: \"a25adb99-ccc9-4dfe-ab8a-e07a082fa0e3\") " pod="openshift-must-gather-s4bld/perf-node-gather-daemonset-blfc8" Jul 10 02:57:39.815763 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:39.815537 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a25adb99-ccc9-4dfe-ab8a-e07a082fa0e3-podres\") pod \"perf-node-gather-daemonset-blfc8\" (UID: \"a25adb99-ccc9-4dfe-ab8a-e07a082fa0e3\") " pod="openshift-must-gather-s4bld/perf-node-gather-daemonset-blfc8" Jul 10 02:57:39.815763 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:39.815552 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a25adb99-ccc9-4dfe-ab8a-e07a082fa0e3-lib-modules\") pod \"perf-node-gather-daemonset-blfc8\" (UID: \"a25adb99-ccc9-4dfe-ab8a-e07a082fa0e3\") " pod="openshift-must-gather-s4bld/perf-node-gather-daemonset-blfc8" Jul 10 02:57:39.824935 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:39.824903 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7thf\" (UniqueName: \"kubernetes.io/projected/a25adb99-ccc9-4dfe-ab8a-e07a082fa0e3-kube-api-access-f7thf\") pod \"perf-node-gather-daemonset-blfc8\" (UID: \"a25adb99-ccc9-4dfe-ab8a-e07a082fa0e3\") " pod="openshift-must-gather-s4bld/perf-node-gather-daemonset-blfc8" Jul 10 02:57:39.956307 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:39.956219 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s4bld/perf-node-gather-daemonset-blfc8" Jul 10 02:57:40.088360 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:40.088336 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s4bld/perf-node-gather-daemonset-blfc8"] Jul 10 02:57:40.089933 ip-10-0-130-65 kubenswrapper[2566]: W0710 02:57:40.089907 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda25adb99_ccc9_4dfe_ab8a_e07a082fa0e3.slice/crio-60dfd558f1e117f75e17801be8daccec0d699b06f969523e17a9a6bba3ea819e WatchSource:0}: Error finding container 60dfd558f1e117f75e17801be8daccec0d699b06f969523e17a9a6bba3ea819e: Status 404 returned error can't find the container with id 60dfd558f1e117f75e17801be8daccec0d699b06f969523e17a9a6bba3ea819e Jul 10 02:57:40.347956 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:40.347924 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s4bld/perf-node-gather-daemonset-blfc8" event={"ID":"a25adb99-ccc9-4dfe-ab8a-e07a082fa0e3","Type":"ContainerStarted","Data":"08cb38542e0951dbe06e72c2f212dc57768eaf18acceb590a178042e8e998201"} Jul 10 02:57:40.348128 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:40.347979 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s4bld/perf-node-gather-daemonset-blfc8" event={"ID":"a25adb99-ccc9-4dfe-ab8a-e07a082fa0e3","Type":"ContainerStarted","Data":"60dfd558f1e117f75e17801be8daccec0d699b06f969523e17a9a6bba3ea819e"} Jul 10 02:57:40.348128 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:40.348005 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-s4bld/perf-node-gather-daemonset-blfc8" Jul 10 02:57:40.369948 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:40.369894 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-s4bld/perf-node-gather-daemonset-blfc8" podStartSLOduration=1.369876456 podStartE2EDuration="1.369876456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 02:57:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:57:40.368426569 +0000 UTC m=+2991.944703676" watchObservedRunningTime="2026-07-10 02:57:40.369876456 +0000 UTC m=+2991.946153542" Jul 10 02:57:41.109372 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:41.109338 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-686777b996-krg22_ca21095f-2156-4769-851c-c06475e23f85/console/0.log" Jul 10 02:57:41.144026 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:41.144000 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-r7zjf_86b9e286-1e9b-43dd-9ff4-ca61dce4bc6c/download-server/0.log" Jul 10 02:57:41.726862 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:41.726836 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-fn4xn_1cddfa1b-4e04-4165-99ff-24071eb16422/volume-data-source-validator/0.log" Jul 10 02:57:42.869408 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:42.869384 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pl7t5_9ce933c0-a5ae-4f64-83af-ec2eef8371b0/dns/0.log" Jul 10 02:57:42.893154 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:42.893133 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pl7t5_9ce933c0-a5ae-4f64-83af-ec2eef8371b0/kube-rbac-proxy/0.log" Jul 10 02:57:43.024949 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:43.024918 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xgndm_8ccffaf3-b29f-4f58-ba37-5ab6f4992d19/dns-node-resolver/0.log" Jul 10 02:57:43.688682 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:43.688654 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8t46j_86176d1c-3214-46c1-91fe-94f087e0bda4/node-ca/0.log" Jul 10 02:57:44.844786 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:44.844756 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-9ts97_49d07846-4ea0-496e-843b-90a8920ea7bc/discovery/0.log" Jul 10 02:57:45.111505 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:45.111434 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6cd68c6448-czfxh_82ce150d-af95-4650-a996-fbe49696196f/router/0.log" Jul 10 02:57:45.869365 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:45.869339 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-r7nm4_2ef36e20-1fdf-4327-bf59-80abf4dfb7bd/serve-healthcheck-canary/0.log" Jul 10 02:57:46.363578 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:46.363551 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-s4bld/perf-node-gather-daemonset-blfc8" Jul 10 02:57:46.428535 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:46.428510 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2w28f_7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8/kube-rbac-proxy/0.log" Jul 10 02:57:46.452500 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:46.452471 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2w28f_7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8/exporter/0.log" Jul 10 02:57:46.477128 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:46.477103 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2w28f_7a7d4a6f-04a7-4c56-b1b6-fe9d77f9a2a8/extractor/0.log" Jul 10 02:57:49.671022 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:49.670985 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-78b9cbb96-rl82t_d85d3688-f7b6-4739-a498-33484bd9b45f/manager/0.log" Jul 10 02:57:49.690994 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:49.690969 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-78b9cbb96-rl82t_d85d3688-f7b6-4739-a498-33484bd9b45f/copy-manifests/0.log" Jul 10 02:57:50.241875 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:50.241845 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-9ts97_49d07846-4ea0-496e-843b-90a8920ea7bc/discovery/0.log" Jul 10 02:57:50.431216 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:50.431180 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6cd68c6448-czfxh_82ce150d-af95-4650-a996-fbe49696196f/router/0.log" Jul 10 02:57:50.987334 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:50.987307 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-58d7cbdf66-2v8tx_4ff2f2ed-7d54-46f5-98ab-0a99f29ffe09/manager/0.log" Jul 10 02:57:51.819821 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:51.819790 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-9ts97_49d07846-4ea0-496e-843b-90a8920ea7bc/discovery/0.log" Jul 10 02:57:52.016912 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:52.016883 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6cd68c6448-czfxh_82ce150d-af95-4650-a996-fbe49696196f/router/0.log" Jul 10 02:57:56.031537 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:56.031506 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-gsh8k_c3de8f05-0340-4da3-b0a6-66534889f709/migrator/0.log" Jul 10 02:57:56.054857 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:56.054831 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-gsh8k_c3de8f05-0340-4da3-b0a6-66534889f709/graceful-termination/0.log" Jul 10 02:57:56.453408 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:56.453333 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-whpts_54ca438c-c22e-45df-8aa2-f653453022b0/kube-storage-version-migrator-operator/1.log" Jul 10 02:57:56.454271 ip-10-0-130-65 kubenswrapper[2566]: I0710 02:57:56.454254 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-whpts_54ca438c-c22e-45df-8aa2-f653453022b0/kube-storage-version-migrator-operator/0.log"