Jul 20 21:00:28.784190 ip-10-0-136-32 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 20 21:00:28.784200 ip-10-0-136-32 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 20 21:00:28.784207 ip-10-0-136-32 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 20 21:00:28.784498 ip-10-0-136-32 systemd[1]: Failed to start Kubernetes Kubelet. Jul 20 21:00:38.944367 ip-10-0-136-32 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 20 21:00:38.944387 ip-10-0-136-32 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 1cb6ade7b02c4597bdba13e6dbf9c2fd -- Jul 20 21:02:38.731059 ip-10-0-136-32 systemd[1]: Starting Kubernetes Kubelet... Jul 20 21:02:39.288460 ip-10-0-136-32 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 20 21:02:39.288460 ip-10-0-136-32 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 20 21:02:39.288460 ip-10-0-136-32 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 20 21:02:39.288460 ip-10-0-136-32 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 20 21:02:39.288460 ip-10-0-136-32 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 20 21:02:39.288460 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.194515 2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 20 21:02:39.288460 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198513 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 21:02:39.288460 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198526 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 21:02:39.288460 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198530 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 21:02:39.288460 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198534 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 21:02:39.288460 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198538 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 21:02:39.288460 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198541 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 21:02:39.288460 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198545 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 21:02:39.288460 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198548 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 21:02:39.259479 ip-10-0-136-32 systemd[1]: Started Kubernetes Kubelet. Jul 20 21:02:39.290179 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198551 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 21:02:39.290179 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198555 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 21:02:39.290179 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198560 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 21:02:39.290179 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198566 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 21:02:39.290179 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198569 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 21:02:39.290179 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198572 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 21:02:39.290179 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198575 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 21:02:39.290179 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198580 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 21:02:39.290179 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198584 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 21:02:39.290179 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198588 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 21:02:39.290179 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198592 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 21:02:39.290179 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198597 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 21:02:39.290179 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198599 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 21:02:39.290179 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198607 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 21:02:39.290179 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198611 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 21:02:39.290179 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198615 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 21:02:39.290179 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198619 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 21:02:39.290179 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198622 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 21:02:39.292406 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198628 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 21:02:39.292406 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198632 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 21:02:39.292406 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198635 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 21:02:39.292406 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198638 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 21:02:39.292406 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198643 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 21:02:39.292406 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198647 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 21:02:39.292406 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198652 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 21:02:39.292406 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198655 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 21:02:39.292406 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198658 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 21:02:39.292406 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198661 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 21:02:39.292406 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198664 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 21:02:39.292406 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198667 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 21:02:39.292406 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198669 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 21:02:39.292406 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198672 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 21:02:39.292406 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198675 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 21:02:39.292406 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198678 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 21:02:39.292406 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198681 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 21:02:39.292406 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198683 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 21:02:39.292406 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198687 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 21:02:39.293057 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198690 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 21:02:39.293057 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198693 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 21:02:39.293057 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198697 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 21:02:39.293057 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198700 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 21:02:39.293057 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198703 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 21:02:39.293057 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198706 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 21:02:39.293057 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198709 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 21:02:39.293057 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198712 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 21:02:39.293057 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198714 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 21:02:39.293057 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198717 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 21:02:39.293057 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198721 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 21:02:39.293057 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198725 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 21:02:39.293057 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198728 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 21:02:39.293057 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198732 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 21:02:39.293057 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198735 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 21:02:39.293057 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198737 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 21:02:39.293057 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198739 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 21:02:39.293057 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198743 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 21:02:39.293057 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198746 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 20 21:02:39.293057 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198749 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 21:02:39.299824 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198751 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 21:02:39.299824 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198755 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 21:02:39.299824 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198758 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 21:02:39.299824 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198760 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 21:02:39.299824 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198763 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 21:02:39.299824 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198783 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 21:02:39.299824 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198787 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 21:02:39.299824 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198790 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 21:02:39.299824 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198793 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 21:02:39.299824 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198796 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 21:02:39.299824 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198798 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 21:02:39.299824 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198802 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 21:02:39.299824 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198806 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 21:02:39.299824 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198808 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 21:02:39.299824 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198813 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 21:02:39.299824 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198817 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 21:02:39.299824 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198820 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 21:02:39.299824 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198823 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 21:02:39.299824 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198826 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 21:02:39.299824 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198829 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 21:02:39.300668 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198831 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 21:02:39.300668 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198834 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 21:02:39.300668 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198838 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 21:02:39.300668 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198840 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 21:02:39.300668 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198843 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 21:02:39.300668 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198845 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 21:02:39.300668 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198849 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 21:02:39.300668 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198852 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 21:02:39.300668 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198855 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 21:02:39.300668 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198857 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 21:02:39.300668 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198861 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 21:02:39.300668 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198866 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 21:02:39.300668 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198869 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 21:02:39.300668 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.198872 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 21:02:39.300668 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199621 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 21:02:39.300668 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199628 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 21:02:39.300668 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199632 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 21:02:39.300668 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199635 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 21:02:39.300668 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199638 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 21:02:39.300668 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199642 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 21:02:39.301339 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199646 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 21:02:39.301339 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199649 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 21:02:39.301339 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199652 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 21:02:39.301339 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199655 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 21:02:39.301339 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199658 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 21:02:39.301339 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199661 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 21:02:39.301339 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199664 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 21:02:39.301339 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199668 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 21:02:39.301339 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199672 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 21:02:39.301339 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199675 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 21:02:39.301339 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199680 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 21:02:39.301339 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199682 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 21:02:39.301339 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199685 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 21:02:39.301339 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199688 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 21:02:39.301339 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199692 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 21:02:39.301339 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199695 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 21:02:39.301339 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199698 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 21:02:39.301339 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199702 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 21:02:39.301339 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199704 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 21:02:39.301339 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199707 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 21:02:39.302091 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199709 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 21:02:39.302091 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199713 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 21:02:39.302091 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199716 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 21:02:39.302091 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199719 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 21:02:39.302091 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199724 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 21:02:39.302091 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199730 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 21:02:39.302091 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199734 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 21:02:39.302091 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199738 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 21:02:39.302091 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199741 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 21:02:39.302091 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199743 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 21:02:39.302091 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199747 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 21:02:39.302091 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199749 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 21:02:39.302091 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199752 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 21:02:39.302091 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199756 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 21:02:39.302091 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199759 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 21:02:39.302091 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199762 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 21:02:39.302091 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199765 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 21:02:39.302091 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199783 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 21:02:39.302091 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199786 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 21:02:39.302812 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199790 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 21:02:39.302812 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199795 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 21:02:39.302812 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199797 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 21:02:39.302812 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199801 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 21:02:39.302812 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199805 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 21:02:39.302812 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199808 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 21:02:39.302812 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199810 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 21:02:39.302812 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199814 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 21:02:39.302812 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199817 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 21:02:39.302812 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199819 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 21:02:39.302812 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199822 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 21:02:39.302812 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199826 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 21:02:39.302812 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199828 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 21:02:39.302812 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199831 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 21:02:39.302812 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199834 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 21:02:39.302812 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199837 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 21:02:39.302812 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199840 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 20 21:02:39.302812 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199843 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 21:02:39.302812 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199846 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 21:02:39.302812 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199850 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 21:02:41.244910 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199853 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 21:02:41.244910 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199856 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 21:02:41.244910 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199859 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 21:02:41.244910 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199862 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 21:02:41.244910 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199865 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 21:02:41.244910 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199867 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 21:02:41.244910 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199871 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 21:02:41.244910 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199874 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 21:02:41.244910 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199877 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 21:02:41.244910 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199881 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 21:02:41.244910 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199884 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 21:02:41.244910 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199888 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 21:02:41.244910 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199890 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 21:02:41.244910 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199894 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 21:02:41.244910 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199898 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 21:02:41.244910 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199900 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 21:02:41.244910 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199904 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 21:02:41.244910 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199907 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 21:02:41.244910 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199910 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 21:02:41.244910 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199913 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 21:02:41.260009 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199917 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 21:02:41.260009 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199920 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 21:02:41.260009 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199922 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 21:02:41.260009 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199925 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 21:02:41.260009 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199928 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 21:02:41.260009 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199931 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 21:02:41.260009 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199933 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 21:02:41.260009 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199937 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 21:02:41.260009 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199940 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 21:02:41.260009 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199943 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 21:02:41.260009 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199947 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 21:02:41.260009 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199951 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 21:02:41.260009 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199954 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 21:02:41.260009 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.199956 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 21:02:41.260009 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200418 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 21:02:41.260009 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200427 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 21:02:41.260009 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200431 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 21:02:41.260009 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200434 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 21:02:41.260009 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200438 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 21:02:41.260009 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200441 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 21:02:41.261068 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200444 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 21:02:41.261068 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200450 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 21:02:41.261068 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200453 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 21:02:41.261068 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200457 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 21:02:41.261068 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200460 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 21:02:41.261068 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200465 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 21:02:41.261068 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200469 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 21:02:41.261068 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200474 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 21:02:41.261068 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200478 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 21:02:41.261068 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200481 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 21:02:41.261068 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200485 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 21:02:41.261068 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200488 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 21:02:41.261068 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200491 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 21:02:41.261068 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200495 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 21:02:41.261068 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200499 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 21:02:41.261068 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200502 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 21:02:41.261068 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200505 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 21:02:41.261068 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200509 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 21:02:41.261068 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200511 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 21:02:41.261923 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200514 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 21:02:41.261923 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200517 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 21:02:41.261923 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200521 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 21:02:41.261923 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200524 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 21:02:41.261923 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200526 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 21:02:41.261923 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200532 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 21:02:41.261923 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200535 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 21:02:41.261923 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200537 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 21:02:41.261923 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200540 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 21:02:41.261923 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200544 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 21:02:41.261923 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200547 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 21:02:41.261923 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200549 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 21:02:41.261923 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200553 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 21:02:41.261923 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200555 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 21:02:41.261923 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200559 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 21:02:41.261923 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200562 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 21:02:41.261923 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200565 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 21:02:41.261923 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200567 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 21:02:41.261923 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200570 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 21:02:41.261923 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200573 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 21:02:41.262964 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200577 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 21:02:41.262964 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200582 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 21:02:41.262964 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200586 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 21:02:41.262964 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200588 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 21:02:41.262964 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200591 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 21:02:41.262964 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200593 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 21:02:41.262964 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200597 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 21:02:41.262964 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200600 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 21:02:41.262964 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200602 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 21:02:41.262964 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200605 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 21:02:41.262964 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200608 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 21:02:41.262964 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200611 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 21:02:41.262964 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200613 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 21:02:41.262964 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200616 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 21:02:41.262964 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200620 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 21:02:41.262964 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200623 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 21:02:41.262964 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200626 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 21:02:41.262964 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200632 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 21:02:41.262964 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200635 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 21:02:41.262964 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200638 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 21:02:41.264034 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200641 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 21:02:41.264034 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200645 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 21:02:41.264034 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200648 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 21:02:41.264034 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200650 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 21:02:41.264034 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200654 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 21:02:41.264034 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200657 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 21:02:41.264034 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200661 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 21:02:41.264034 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200665 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 21:02:41.264034 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200669 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 21:02:41.264034 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200672 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 21:02:41.264034 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200675 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 21:02:41.264034 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200679 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 21:02:41.264034 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200682 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 21:02:41.264034 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200685 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 21:02:41.264034 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200689 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 21:02:41.264034 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200692 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 21:02:41.264034 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200695 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 21:02:41.264034 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200697 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 21:02:41.264034 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200701 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 21:02:41.264034 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200704 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 20 21:02:41.265122 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200706 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 21:02:41.265122 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200709 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 21:02:41.265122 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200712 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 21:02:41.265122 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200715 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 21:02:41.265122 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200717 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 21:02:41.265122 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200720 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 21:02:41.265122 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200723 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 21:02:41.265122 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200726 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 21:02:41.265122 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200729 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 21:02:41.265122 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200736 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 21:02:41.265122 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200739 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 21:02:41.265122 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200742 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 21:02:41.265122 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200746 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 21:02:41.265122 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.200749 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 21:02:41.265122 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.200831 2562 flags.go:64] FLAG: --address="0.0.0.0" Jul 20 21:02:41.265122 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.200856 2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 20 21:02:41.265122 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.200864 2562 flags.go:64] FLAG: --anonymous-auth="true" Jul 20 21:02:41.265122 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.200871 2562 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 20 21:02:41.265122 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.200877 2562 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 20 21:02:41.265122 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.200882 2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 20 21:02:41.265940 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.200887 2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 20 21:02:41.265940 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.200890 2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 20 21:02:41.265940 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.200895 2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 20 21:02:41.265940 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.200898 2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 20 21:02:41.265940 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.200901 2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 20 21:02:41.265940 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.200905 2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 20 21:02:41.265940 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.200909 2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 20 21:02:41.265940 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.200922 2562 flags.go:64] FLAG: --cgroup-root="" Jul 20 21:02:41.265940 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.200926 2562 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 20 21:02:41.265940 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.200930 2562 flags.go:64] FLAG: --client-ca-file="" Jul 20 21:02:41.265940 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.200934 2562 flags.go:64] FLAG: --cloud-provider="external" Jul 20 21:02:41.265940 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.200937 2562 flags.go:64] FLAG: --cluster-dns="[]" Jul 20 21:02:41.265940 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.200943 2562 flags.go:64] FLAG: --cluster-domain="" Jul 20 21:02:41.265940 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.200946 2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 20 21:02:41.265940 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.200949 2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 20 21:02:41.265940 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.200954 2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 20 21:02:41.265940 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.200958 2562 flags.go:64] FLAG: --container-log-max-files="5" Jul 20 21:02:41.265940 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.200962 2562 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 20 21:02:41.265940 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.200966 2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 20 21:02:41.265940 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.200970 2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 20 21:02:41.265940 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.200973 2562 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 20 21:02:41.265940 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.200976 2562 flags.go:64] FLAG: --contention-profiling="false" Jul 20 21:02:41.265940 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.200981 2562 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 20 21:02:41.265940 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.200985 2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 20 21:02:41.266958 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.200989 2562 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 20 21:02:41.266958 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.200992 2562 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 20 21:02:41.266958 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.200996 2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 20 21:02:41.266958 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201001 2562 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 20 21:02:41.266958 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201004 2562 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 20 21:02:41.266958 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201007 2562 flags.go:64] FLAG: --enable-load-reader="false" Jul 20 21:02:41.266958 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201018 2562 flags.go:64] FLAG: --enable-server="true" Jul 20 21:02:41.266958 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201022 2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 20 21:02:41.266958 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201027 2562 flags.go:64] FLAG: --event-burst="100" Jul 20 21:02:41.266958 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201030 2562 flags.go:64] FLAG: --event-qps="50" Jul 20 21:02:41.266958 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201033 2562 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 20 21:02:41.266958 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201037 2562 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 20 21:02:41.266958 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201040 2562 flags.go:64] FLAG: --eviction-hard="" Jul 20 21:02:41.266958 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201045 2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 20 21:02:41.266958 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201048 2562 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 20 21:02:41.266958 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201052 2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 20 21:02:41.266958 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201056 2562 flags.go:64] FLAG: --eviction-soft="" Jul 20 21:02:41.266958 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201060 2562 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 20 21:02:41.266958 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201063 2562 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 20 21:02:41.266958 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201066 2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 20 21:02:41.266958 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201070 2562 flags.go:64] FLAG: --experimental-mounter-path="" Jul 20 21:02:41.266958 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201074 2562 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 20 21:02:41.266958 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201078 2562 flags.go:64] FLAG: --fail-swap-on="true" Jul 20 21:02:41.266958 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201082 2562 flags.go:64] FLAG: --feature-gates="" Jul 20 21:02:41.266958 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201086 2562 flags.go:64] FLAG: --file-check-frequency="20s" Jul 20 21:02:41.267998 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201091 2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 20 21:02:41.267998 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201094 2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 20 21:02:41.267998 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201097 2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 20 21:02:41.267998 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201101 2562 flags.go:64] FLAG: --healthz-port="10248" Jul 20 21:02:41.267998 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201105 2562 flags.go:64] FLAG: --help="false" Jul 20 21:02:41.267998 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201110 2562 flags.go:64] FLAG: --hostname-override="ip-10-0-136-32.ec2.internal" Jul 20 21:02:41.267998 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201115 2562 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 20 21:02:41.267998 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201118 2562 flags.go:64] FLAG: --http-check-frequency="20s" Jul 20 21:02:41.267998 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201121 2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 20 21:02:41.267998 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201125 2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 20 21:02:41.267998 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201129 2562 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 20 21:02:41.267998 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201132 2562 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 20 21:02:41.267998 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201135 2562 flags.go:64] FLAG: --image-service-endpoint="" Jul 20 21:02:41.267998 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201141 2562 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 20 21:02:41.267998 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201144 2562 flags.go:64] FLAG: --kube-api-burst="100" Jul 20 21:02:41.267998 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201147 2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 20 21:02:41.267998 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201151 2562 flags.go:64] FLAG: --kube-api-qps="50" Jul 20 21:02:41.267998 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201154 2562 flags.go:64] FLAG: --kube-reserved="" Jul 20 21:02:41.267998 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201158 2562 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 20 21:02:41.267998 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201162 2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 20 21:02:41.267998 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201166 2562 flags.go:64] FLAG: --kubelet-cgroups="" Jul 20 21:02:41.267998 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201169 2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 20 21:02:41.267998 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201173 2562 flags.go:64] FLAG: --lock-file="" Jul 20 21:02:41.267998 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201178 2562 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 20 21:02:41.268812 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201181 2562 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 20 21:02:41.268812 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201192 2562 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 20 21:02:41.268812 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201198 2562 flags.go:64] FLAG: --log-json-split-stream="false" Jul 20 21:02:41.268812 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201201 2562 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 20 21:02:41.268812 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201204 2562 flags.go:64] FLAG: --log-text-split-stream="false" Jul 20 21:02:41.268812 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201208 2562 flags.go:64] FLAG: --logging-format="text" Jul 20 21:02:41.268812 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201212 2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 20 21:02:41.268812 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201215 2562 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 20 21:02:41.268812 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201218 2562 flags.go:64] FLAG: --manifest-url="" Jul 20 21:02:41.268812 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201222 2562 flags.go:64] FLAG: --manifest-url-header="" Jul 20 21:02:41.268812 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201227 2562 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 20 21:02:41.268812 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201232 2562 flags.go:64] FLAG: --max-open-files="1000000" Jul 20 21:02:41.268812 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201236 2562 flags.go:64] FLAG: --max-pods="110" Jul 20 21:02:41.268812 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201240 2562 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 20 21:02:41.268812 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201244 2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 20 21:02:41.268812 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201247 2562 flags.go:64] FLAG: --memory-manager-policy="None" Jul 20 21:02:41.268812 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201250 2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 20 21:02:41.268812 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201254 2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 20 21:02:41.268812 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201258 2562 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 20 21:02:41.268812 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201261 2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 20 21:02:41.268812 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201270 2562 flags.go:64] FLAG: --node-status-max-images="50" Jul 20 21:02:41.268812 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201275 2562 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 20 21:02:41.268812 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201279 2562 flags.go:64] FLAG: --oom-score-adj="-999" Jul 20 21:02:41.268812 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201282 2562 flags.go:64] FLAG: --pod-cidr="" Jul 20 21:02:41.269472 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201285 2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 20 21:02:41.269472 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201292 2562 flags.go:64] FLAG: --pod-manifest-path="" Jul 20 21:02:41.269472 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201295 2562 flags.go:64] FLAG: --pod-max-pids="-1" Jul 20 21:02:41.269472 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201300 2562 flags.go:64] FLAG: --pods-per-core="0" Jul 20 21:02:41.269472 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201304 2562 flags.go:64] FLAG: --port="10250" Jul 20 21:02:41.269472 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201307 2562 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 20 21:02:41.269472 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201310 2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06a792d6ae144e738" Jul 20 21:02:41.269472 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201315 2562 flags.go:64] FLAG: --qos-reserved="" Jul 20 21:02:41.269472 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201319 2562 flags.go:64] FLAG: --read-only-port="10255" Jul 20 21:02:41.269472 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201322 2562 flags.go:64] FLAG: --register-node="true" Jul 20 21:02:41.269472 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201326 2562 flags.go:64] FLAG: --register-with-taints="" Jul 20 21:02:41.269472 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201330 2562 flags.go:64] FLAG: --registry-burst="10" Jul 20 21:02:41.269472 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201334 2562 flags.go:64] FLAG: --registry-qps="5" Jul 20 21:02:41.269472 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201338 2562 flags.go:64] FLAG: --reserved-cpus="" Jul 20 21:02:41.269472 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201341 2562 flags.go:64] FLAG: --reserved-memory="" Jul 20 21:02:41.269472 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201346 2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 20 21:02:41.269472 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201350 2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 20 21:02:41.269472 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201353 2562 flags.go:64] FLAG: --rotate-certificates="false" Jul 20 21:02:41.269472 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201356 2562 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 20 21:02:41.269472 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201360 2562 flags.go:64] FLAG: --runonce="false" Jul 20 21:02:41.269472 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201363 2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 20 21:02:41.269472 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201369 2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 20 21:02:41.269472 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201373 2562 flags.go:64] FLAG: --seccomp-default="false" Jul 20 21:02:41.269472 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201375 2562 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 20 21:02:41.269472 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201379 2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 20 21:02:41.270234 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201382 2562 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 20 21:02:41.270234 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201385 2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 20 21:02:41.270234 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201388 2562 flags.go:64] FLAG: --storage-driver-password="root" Jul 20 21:02:41.270234 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201391 2562 flags.go:64] FLAG: --storage-driver-secure="false" Jul 20 21:02:41.270234 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201394 2562 flags.go:64] FLAG: --storage-driver-table="stats" Jul 20 21:02:41.270234 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201398 2562 flags.go:64] FLAG: --storage-driver-user="root" Jul 20 21:02:41.270234 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201402 2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 20 21:02:41.270234 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201405 2562 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 20 21:02:41.270234 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201408 2562 flags.go:64] FLAG: --system-cgroups="" Jul 20 21:02:41.270234 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201412 2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 20 21:02:41.270234 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201418 2562 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 20 21:02:41.270234 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201421 2562 flags.go:64] FLAG: --tls-cert-file="" Jul 20 21:02:41.270234 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201424 2562 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 20 21:02:41.270234 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201429 2562 flags.go:64] FLAG: --tls-min-version="" Jul 20 21:02:41.270234 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201432 2562 flags.go:64] FLAG: --tls-private-key-file="" Jul 20 21:02:41.270234 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201435 2562 flags.go:64] FLAG: --topology-manager-policy="none" Jul 20 21:02:41.270234 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201439 2562 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 20 21:02:41.270234 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201442 2562 flags.go:64] FLAG: --topology-manager-scope="container" Jul 20 21:02:41.270234 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201446 2562 flags.go:64] FLAG: --v="2" Jul 20 21:02:41.270234 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201451 2562 flags.go:64] FLAG: --version="false" Jul 20 21:02:41.270234 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201455 2562 flags.go:64] FLAG: --vmodule="" Jul 20 21:02:41.270234 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201460 2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 20 21:02:41.270234 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.201463 2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 20 21:02:41.270234 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201588 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 21:02:41.270234 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201592 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 21:02:41.270896 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201595 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 21:02:41.270896 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201598 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 21:02:41.270896 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201600 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 21:02:41.270896 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201606 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 21:02:41.270896 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201609 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 21:02:41.270896 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201612 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 21:02:41.270896 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201614 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 21:02:41.270896 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201618 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 21:02:41.270896 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201621 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 21:02:41.270896 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201624 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 21:02:41.270896 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201627 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 21:02:41.270896 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201631 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 21:02:41.270896 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201636 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 21:02:41.270896 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201640 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 21:02:41.270896 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201643 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 21:02:41.270896 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201645 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 21:02:41.270896 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201648 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 21:02:41.270896 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201651 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 21:02:41.270896 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201654 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 21:02:41.270896 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201658 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 21:02:41.271491 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201661 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 21:02:41.271491 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201664 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 21:02:41.271491 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201667 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 21:02:41.271491 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201671 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 21:02:41.271491 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201677 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 21:02:41.271491 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201680 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 21:02:41.271491 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201683 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 21:02:41.271491 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201687 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 21:02:41.271491 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201689 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 21:02:41.271491 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201692 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 21:02:41.271491 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201695 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 21:02:41.271491 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201699 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 21:02:41.271491 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201701 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 21:02:41.271491 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201704 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 21:02:41.271491 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201706 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 21:02:41.271491 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201712 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 21:02:41.271491 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201715 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 21:02:41.271491 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201717 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 21:02:41.271491 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201721 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 21:02:41.271491 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201724 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 21:02:41.272330 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201726 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 21:02:41.272330 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201729 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 21:02:41.272330 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201733 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 21:02:41.272330 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201736 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 21:02:41.272330 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201742 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 21:02:41.272330 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201747 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 21:02:41.272330 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201751 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 20 21:02:41.272330 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201754 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 21:02:41.272330 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201758 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 21:02:41.272330 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201760 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 21:02:41.272330 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201763 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 21:02:41.272330 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201779 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 21:02:41.272330 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201782 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 21:02:41.272330 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201785 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 21:02:41.272330 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201789 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 21:02:41.272330 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201791 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 21:02:41.272330 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201794 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 21:02:41.272330 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201797 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 21:02:41.272330 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201800 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 21:02:41.272879 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201802 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 21:02:41.272879 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201805 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 21:02:41.272879 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201809 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 21:02:41.272879 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201812 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 21:02:41.272879 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201815 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 21:02:41.272879 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201817 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 21:02:41.272879 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201820 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 21:02:41.272879 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201824 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 21:02:41.272879 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201827 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 21:02:41.272879 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201830 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 21:02:41.272879 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201834 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 21:02:41.272879 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201837 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 21:02:41.272879 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201840 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 21:02:41.272879 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201843 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 21:02:41.272879 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201847 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 21:02:41.272879 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201850 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 21:02:41.272879 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201852 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 21:02:41.272879 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201857 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 21:02:41.272879 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201860 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 21:02:41.272879 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201862 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 21:02:41.273533 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201865 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 21:02:41.273533 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201869 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 21:02:41.273533 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201873 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 21:02:41.273533 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201876 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 21:02:41.273533 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201880 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 21:02:41.273533 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201883 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 21:02:41.273533 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201886 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 21:02:41.273533 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201890 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 21:02:41.273533 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201893 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 21:02:41.273533 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201896 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 21:02:41.273533 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201898 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 21:02:41.273533 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201902 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 21:02:41.273533 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201905 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 21:02:41.273533 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201908 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 21:02:41.273533 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201911 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 21:02:41.273533 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201914 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 21:02:41.273533 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201918 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 21:02:41.273533 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.201920 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 21:02:41.274051 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.202431 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 20 21:02:41.274051 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.209129 2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 20 21:02:41.274051 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.209144 2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 20 21:02:41.274051 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.209163 2562 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 20 21:02:41.274051 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.209168 2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 20 21:02:41.274051 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209231 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 21:02:41.274051 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209236 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 21:02:41.274051 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209240 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 21:02:41.274051 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209243 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 21:02:41.274051 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209246 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 21:02:41.274051 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209249 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 21:02:41.274051 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209252 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 21:02:41.274051 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209255 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 21:02:41.274051 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209258 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 21:02:41.274545 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209261 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 21:02:41.274545 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209264 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 21:02:41.274545 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209267 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 21:02:41.274545 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209270 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 21:02:41.274545 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209273 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 21:02:41.274545 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209276 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 21:02:41.274545 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209279 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 21:02:41.274545 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209281 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 21:02:41.274545 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209284 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 21:02:41.274545 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209287 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 21:02:41.274545 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209290 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 21:02:41.274545 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209296 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 21:02:41.274545 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209298 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 21:02:41.274545 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209301 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 21:02:41.274545 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209305 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 21:02:41.274545 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209308 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 21:02:41.274545 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209310 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 21:02:41.274545 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209313 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 21:02:41.274545 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209316 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 21:02:41.274545 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209318 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 20 21:02:41.275093 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209323 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 21:02:41.275093 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209326 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 21:02:41.275093 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209330 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 21:02:41.275093 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209332 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 21:02:41.275093 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209335 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 21:02:41.275093 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209338 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 21:02:41.275093 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209341 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 21:02:41.275093 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209343 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 21:02:41.275093 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209346 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 21:02:41.275093 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209348 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 21:02:41.275093 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209351 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 21:02:41.275093 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209353 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 21:02:41.275093 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209357 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 21:02:41.275093 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209360 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 21:02:41.275093 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209362 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 21:02:41.275093 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209365 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 21:02:41.275093 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209368 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 21:02:41.275093 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209370 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 21:02:41.275093 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209375 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 21:02:41.393284 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209378 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 21:02:41.393284 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209380 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 21:02:41.393284 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209383 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 21:02:41.393284 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209386 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 21:02:41.393284 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209390 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 21:02:41.393284 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209394 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 21:02:41.393284 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209397 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 21:02:41.393284 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209402 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 21:02:41.393284 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209405 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 21:02:41.393284 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209408 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 21:02:41.393284 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209411 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 21:02:41.393284 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209413 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 21:02:41.393284 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209416 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 21:02:41.393284 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209419 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 21:02:41.393284 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209422 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 21:02:41.393284 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209424 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 21:02:41.393284 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209427 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 21:02:41.393284 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209431 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 21:02:41.393284 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209434 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 21:02:41.393952 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209437 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 21:02:41.393952 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209439 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 21:02:41.393952 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209442 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 21:02:41.393952 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209444 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 21:02:41.393952 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209447 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 21:02:41.393952 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209450 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 21:02:41.393952 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209452 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 21:02:41.393952 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209455 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 21:02:41.393952 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209458 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 21:02:41.393952 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209460 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 21:02:41.393952 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209463 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 21:02:41.393952 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209465 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 21:02:41.393952 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209469 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 21:02:41.393952 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209472 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 21:02:41.393952 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209475 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 21:02:41.393952 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209477 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 21:02:41.393952 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209480 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 21:02:41.393952 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209482 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 21:02:41.393952 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209485 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 21:02:41.393952 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209488 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 21:02:41.394614 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209492 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 21:02:41.394614 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209495 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 21:02:41.394614 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209498 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 21:02:41.394614 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209501 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 21:02:41.394614 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209503 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 21:02:41.394614 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209506 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 21:02:41.394614 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209508 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 21:02:41.394614 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209511 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 21:02:41.394614 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209513 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 21:02:41.394614 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209516 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 21:02:41.394614 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209519 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 21:02:41.394614 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209521 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 21:02:41.394614 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.209526 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 20 21:02:41.394614 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209632 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 21:02:41.394614 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209637 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 21:02:41.395035 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209640 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 21:02:41.395035 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209643 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 21:02:41.395035 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209646 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 21:02:41.395035 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209649 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 21:02:41.395035 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209652 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 21:02:41.395035 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209655 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 21:02:41.395035 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209657 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 21:02:41.395035 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209660 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 21:02:41.395035 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209662 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 21:02:41.395035 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209667 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 20 21:02:41.395035 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209669 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 21:02:41.395035 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209672 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 21:02:41.395035 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209674 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 21:02:41.395035 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209677 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 21:02:41.395035 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209680 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 21:02:41.395035 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209682 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 21:02:41.395035 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209685 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 21:02:41.395035 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209688 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 21:02:41.395035 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209691 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 21:02:41.395035 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209694 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 21:02:41.395661 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209698 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 21:02:41.395661 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209702 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 21:02:41.395661 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209704 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 21:02:41.395661 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209707 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 21:02:41.395661 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209710 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 21:02:41.395661 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209713 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 21:02:41.395661 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209715 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 21:02:41.395661 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209718 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 21:02:41.395661 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209720 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 21:02:41.395661 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209723 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 21:02:41.395661 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209726 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 21:02:41.395661 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209728 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 21:02:41.395661 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209731 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 21:02:41.395661 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209733 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 21:02:41.395661 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209735 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 21:02:41.395661 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209738 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 21:02:41.395661 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209741 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 21:02:41.395661 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209743 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 21:02:41.395661 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209746 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 21:02:41.395661 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209749 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 21:02:41.396660 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209752 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 21:02:41.396660 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209756 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 21:02:41.396660 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209759 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 21:02:41.396660 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209763 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 21:02:41.396660 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209782 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 21:02:41.396660 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209786 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 21:02:41.396660 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209789 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 21:02:41.396660 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209792 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 21:02:41.396660 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209795 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 21:02:41.396660 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209797 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 21:02:41.396660 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209802 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 21:02:41.396660 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209804 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 21:02:41.396660 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209807 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 21:02:41.396660 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209810 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 21:02:41.396660 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209812 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 21:02:41.396660 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209815 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 21:02:41.396660 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209817 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 21:02:41.396660 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209820 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 21:02:41.396660 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209822 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 21:02:41.397208 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209825 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 21:02:41.397208 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209827 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 21:02:41.397208 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209830 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 21:02:41.397208 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209833 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 21:02:41.397208 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209836 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 21:02:41.397208 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209838 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 21:02:41.397208 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209841 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 21:02:41.397208 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209843 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 21:02:41.397208 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209846 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 21:02:41.397208 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209848 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 21:02:41.397208 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209851 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 21:02:41.397208 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209853 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 21:02:41.397208 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209856 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 21:02:41.397208 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209858 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 21:02:41.397208 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209863 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 21:02:41.397208 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209865 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 21:02:41.397208 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209868 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 21:02:41.397208 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209870 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 21:02:41.397208 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209873 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 21:02:41.397208 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209875 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 21:02:41.397804 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209878 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 21:02:41.397804 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209880 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 21:02:41.397804 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209883 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 21:02:41.397804 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209887 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 21:02:41.397804 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209890 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 21:02:41.397804 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209893 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 21:02:41.397804 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209896 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 21:02:41.397804 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209898 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 21:02:41.397804 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209901 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 21:02:41.397804 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209904 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 21:02:41.397804 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209906 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 21:02:41.397804 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209909 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 21:02:41.397804 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209911 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 21:02:41.397804 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209914 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 21:02:41.397804 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209917 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 21:02:41.397804 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209919 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 21:02:41.397804 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209922 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 21:02:41.397804 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:39.209924 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 21:02:41.398584 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.209929 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 20 21:02:41.398584 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.210024 2562 server.go:962] "Client rotation is on, will bootstrap in background" Jul 20 21:02:41.398584 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.211906 2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 20 21:02:41.398584 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.213141 2562 server.go:1019] "Starting client certificate rotation" Jul 20 21:02:41.398584 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.213251 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 20 21:02:41.398584 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.213953 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 20 21:02:41.398584 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.215568 2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 20 21:02:41.398584 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.216675 2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 20 21:02:41.398584 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.227485 2562 log.go:25] "Validated CRI v1 runtime API" Jul 20 21:02:41.398584 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.230340 2562 log.go:25] "Validated CRI v1 image API" Jul 20 21:02:41.398584 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.231826 2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 20 21:02:41.398584 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.234380 2562 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ab87e349-da65-418e-a3ed-fd3196a0086c:/dev/nvme0n1p3 dab161a7-c76f-41e5-9a1c-eaf9c043c943:/dev/nvme0n1p4] Jul 20 21:02:41.399272 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.234395 2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 20 21:02:41.399454 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.239754 2562 manager.go:216] Machine: {Timestamp:2026-07-20 21:02:39.238430449 +0000 UTC m=+0.374497582 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3108795 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d22bc72cd8c4eb19f80ea6b90b497 SystemUUID:ec2d22bc-72cd-8c4e-b19f-80ea6b90b497 BootID:1cb6ade7-b02c-4597-bdba-13e6dbf9c2fd Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:93:81:ae:81:1d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:93:81:ae:81:1d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9e:71:ac:50:68:76 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 20 21:02:41.399454 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.239860 2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 20 21:02:41.399454 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.239936 2562 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 20 21:02:41.399454 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.242680 2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 20 21:02:41.399454 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.242702 2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-32.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 20 21:02:41.399454 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.243451 2562 topology_manager.go:138] "Creating topology manager with none policy" Jul 20 21:02:41.399454 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.243460 2562 container_manager_linux.go:309] "Creating device plugin manager" Jul 20 21:02:41.399454 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.243468 2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 20 21:02:41.400255 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.243479 2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 20 21:02:41.400255 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.243906 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 20 21:02:41.400255 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.244720 2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 20 21:02:41.400255 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.247908 2562 state_mem.go:36] "Initialized new in-memory state store" Jul 20 21:02:41.400255 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.248046 2562 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 20 21:02:41.400255 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.248083 2562 kubelet.go:486] "Attempting to sync node with API server" Jul 20 21:02:41.400255 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.248099 2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 20 21:02:41.400255 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.248117 2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 20 21:02:41.400255 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.248129 2562 kubelet.go:398] "Adding apiserver pod source" Jul 20 21:02:41.400255 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.248166 2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 20 21:02:41.400255 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.248548 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 20 21:02:41.400255 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.248575 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 20 21:02:41.400255 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.252148 2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 20 21:02:41.400255 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.254105 2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 20 21:02:41.400255 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.254152 2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 20 21:02:41.400255 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.255153 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 20 21:02:41.400255 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.255172 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 20 21:02:41.400255 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.255182 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 20 21:02:41.400255 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.255191 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 20 21:02:41.400255 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.255204 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 20 21:02:41.400255 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.255213 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 20 21:02:41.400255 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.255225 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 20 21:02:41.400255 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.255233 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 20 21:02:41.400255 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.255243 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 20 21:02:41.400255 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.255251 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 20 21:02:41.400255 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.255264 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 20 21:02:41.400255 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.255271 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 20 21:02:41.400255 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.255297 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 20 21:02:41.400255 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.255305 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 20 21:02:41.400255 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.258697 2562 server.go:1268] "Started kubelet" Jul 20 21:02:41.400255 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.259327 2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 20 21:02:41.400255 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.259633 2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 20 21:02:41.400255 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.259964 2562 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 20 21:02:41.400255 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.260795 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-m9vgm" Jul 20 21:02:41.400255 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.261975 2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 20 21:02:41.400255 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.265573 2562 server.go:310] "Adding debug handlers to kubelet server" Jul 20 21:02:41.400255 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.266027 2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-32.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 20 21:02:41.400255 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:39.266215 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 20 21:02:41.400255 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:39.266424 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-32.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 20 21:02:41.401798 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.268550 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-m9vgm" Jul 20 21:02:41.401798 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:39.268666 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-32.ec2.internal.18c41b4a87658cae default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-32.ec2.internal,UID:ip-10-0-136-32.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-32.ec2.internal,},FirstTimestamp:2026-07-20 21:02:39.258668206 +0000 UTC m=+0.394735401,LastTimestamp:2026-07-20 21:02:39.258668206 +0000 UTC m=+0.394735401,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-32.ec2.internal,}" Jul 20 21:02:41.401798 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.272824 2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 20 21:02:41.401798 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:39.272902 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-32.ec2.internal\" not found" Jul 20 21:02:41.401798 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:39.272954 2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 20 21:02:41.401798 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.272965 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 20 21:02:41.401798 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.273356 2562 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 20 21:02:41.401798 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.273369 2562 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 20 21:02:41.401798 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.273430 2562 reconstruct.go:97] "Volume reconstruction finished" Jul 20 21:02:41.401798 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.273429 2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 20 21:02:41.401798 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.273440 2562 reconciler.go:29] "Reconciler: start to sync state" Jul 20 21:02:41.401798 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.273633 2562 factory.go:55] Registering systemd factory Jul 20 21:02:41.401798 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.273648 2562 factory.go:223] Registration of the systemd container factory successfully Jul 20 21:02:41.401798 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.273830 2562 factory.go:153] Registering CRI-O factory Jul 20 21:02:41.401798 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.273842 2562 factory.go:223] Registration of the crio container factory successfully Jul 20 21:02:41.401798 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.273890 2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 20 21:02:41.401798 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.273915 2562 factory.go:103] Registering Raw factory Jul 20 21:02:41.401798 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.273929 2562 manager.go:1196] Started watching for new ooms in manager Jul 20 21:02:41.401798 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.274682 2562 manager.go:349] Starting recovery of all containers Jul 20 21:02:41.401798 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.276149 2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 21:02:41.401798 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:39.280558 2562 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-136-32.ec2.internal\" not found" node="ip-10-0-136-32.ec2.internal" Jul 20 21:02:41.401798 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.285197 2562 manager.go:354] Recovery completed Jul 20 21:02:41.401798 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.293655 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 21:02:41.401798 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.295904 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-32.ec2.internal" event="NodeHasSufficientMemory" Jul 20 21:02:41.401798 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.295929 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-32.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 21:02:41.401798 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.295939 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-32.ec2.internal" event="NodeHasSufficientPID" Jul 20 21:02:41.401798 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.296409 2562 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 20 21:02:41.401798 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.296420 2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 20 21:02:41.401798 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.296441 2562 state_mem.go:36] "Initialized new in-memory state store" Jul 20 21:02:41.401798 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.296423 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 21:02:41.401798 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.297144 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-32.ec2.internal" event="NodeHasSufficientMemory" Jul 20 21:02:41.401798 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.297168 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-32.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 21:02:41.401798 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.297180 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-32.ec2.internal" event="NodeHasSufficientPID" Jul 20 21:02:41.401798 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.297586 2562 policy_none.go:49] "None policy: Start" Jul 20 21:02:41.401798 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.297599 2562 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 20 21:02:41.401798 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.297609 2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 20 21:02:41.537663 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.298675 2562 policy_none.go:47] "Start" Jul 20 21:02:41.537663 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.336327 2562 manager.go:337] "Starting Device Plugin manager" Jul 20 21:02:41.537663 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:39.336501 2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 20 21:02:41.537663 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.336519 2562 server.go:87] "Starting device plugin registration server" Jul 20 21:02:41.537663 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.336717 2562 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 20 21:02:41.537663 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.336728 2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 20 21:02:41.537663 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.336934 2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 20 21:02:41.537663 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.337009 2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 20 21:02:41.537663 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:39.337018 2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 20 21:02:41.537663 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:39.337473 2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 20 21:02:41.537663 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:39.337512 2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-32.ec2.internal\" not found" Jul 20 21:02:41.537663 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:41.247506 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 21:02:41.537663 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:41.249467 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-32.ec2.internal" event="NodeHasSufficientMemory" Jul 20 21:02:41.537663 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:41.249506 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-32.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 21:02:41.537663 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:41.249522 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-32.ec2.internal" event="NodeHasSufficientPID" Jul 20 21:02:41.537663 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:41.249565 2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-32.ec2.internal" Jul 20 21:02:41.537663 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:41.258127 2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-32.ec2.internal" Jul 20 21:02:41.537663 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:41.258164 2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-32.ec2.internal\": node \"ip-10-0-136-32.ec2.internal\" not found" Jul 20 21:02:41.537663 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:41.262306 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 20 21:02:41.537663 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:41.263497 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 20 21:02:41.537663 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:41.263531 2562 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 20 21:02:41.537663 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:41.263560 2562 kubelet.go:2448] "Starting kubelet main sync loop" Jul 20 21:02:41.537663 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:41.263600 2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 20 21:02:41.537663 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:41.265698 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 21:02:41.537663 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:41.281173 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-32.ec2.internal\" not found" Jul 20 21:02:41.537663 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:41.364165 2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-32.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-32.ec2.internal"] Jul 20 21:02:41.537663 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:41.364257 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 21:02:41.537663 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:41.365258 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-32.ec2.internal" event="NodeHasSufficientMemory" Jul 20 21:02:41.537663 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:41.365286 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-32.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 21:02:41.537663 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:41.365299 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-32.ec2.internal" event="NodeHasSufficientPID" Jul 20 21:02:41.537663 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:41.366768 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 21:02:41.537663 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:41.367360 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-32.ec2.internal" Jul 20 21:02:41.537663 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:41.367390 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 21:02:41.537663 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:41.367417 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-32.ec2.internal" event="NodeHasSufficientMemory" Jul 20 21:02:41.537663 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:41.367437 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-32.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 21:02:41.537663 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:41.367447 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-32.ec2.internal" event="NodeHasSufficientPID" Jul 20 21:02:41.537663 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:41.367960 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-32.ec2.internal" event="NodeHasSufficientMemory" Jul 20 21:02:41.539417 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:41.367977 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-32.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 21:02:41.539417 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:41.367986 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-32.ec2.internal" event="NodeHasSufficientPID" Jul 20 21:02:41.539417 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:41.368666 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-32.ec2.internal" Jul 20 21:02:41.539417 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:41.368685 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 21:02:41.539417 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:41.369371 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-32.ec2.internal" event="NodeHasSufficientMemory" Jul 20 21:02:41.539417 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:41.369390 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-32.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 21:02:41.539417 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:41.369400 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-32.ec2.internal" event="NodeHasSufficientPID" Jul 20 21:02:41.539417 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:41.381699 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-32.ec2.internal\" not found" Jul 20 21:02:41.539417 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:41.386007 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/197924819a9b1fbec1e5ecce03cf3186-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-32.ec2.internal\" (UID: \"197924819a9b1fbec1e5ecce03cf3186\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-32.ec2.internal" Jul 20 21:02:41.539417 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:41.386030 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/197924819a9b1fbec1e5ecce03cf3186-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-32.ec2.internal\" (UID: \"197924819a9b1fbec1e5ecce03cf3186\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-32.ec2.internal" Jul 20 21:02:41.539417 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:41.386055 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6ba459730c951195c7a8ffe9d5d50c7d-config\") pod \"kube-apiserver-proxy-ip-10-0-136-32.ec2.internal\" (UID: \"6ba459730c951195c7a8ffe9d5d50c7d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-32.ec2.internal" Jul 20 21:02:41.539417 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:41.400435 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-32.ec2.internal\" not found" node="ip-10-0-136-32.ec2.internal" Jul 20 21:02:41.539417 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:41.405437 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-32.ec2.internal\" not found" node="ip-10-0-136-32.ec2.internal" Jul 20 21:02:41.539417 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:41.482042 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-32.ec2.internal\" not found" Jul 20 21:02:41.539417 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:41.486714 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/197924819a9b1fbec1e5ecce03cf3186-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-32.ec2.internal\" (UID: \"197924819a9b1fbec1e5ecce03cf3186\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-32.ec2.internal" Jul 20 21:02:41.539417 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:41.486735 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/197924819a9b1fbec1e5ecce03cf3186-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-32.ec2.internal\" (UID: \"197924819a9b1fbec1e5ecce03cf3186\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-32.ec2.internal" Jul 20 21:02:41.539417 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:41.486752 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6ba459730c951195c7a8ffe9d5d50c7d-config\") pod \"kube-apiserver-proxy-ip-10-0-136-32.ec2.internal\" (UID: \"6ba459730c951195c7a8ffe9d5d50c7d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-32.ec2.internal" Jul 20 21:02:41.539417 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:41.486776 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6ba459730c951195c7a8ffe9d5d50c7d-config\") pod \"kube-apiserver-proxy-ip-10-0-136-32.ec2.internal\" (UID: \"6ba459730c951195c7a8ffe9d5d50c7d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-32.ec2.internal" Jul 20 21:02:41.539417 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:41.486811 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/197924819a9b1fbec1e5ecce03cf3186-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-32.ec2.internal\" (UID: \"197924819a9b1fbec1e5ecce03cf3186\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-32.ec2.internal" Jul 20 21:02:41.541045 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:41.486819 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/197924819a9b1fbec1e5ecce03cf3186-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-32.ec2.internal\" (UID: \"197924819a9b1fbec1e5ecce03cf3186\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-32.ec2.internal" Jul 20 21:02:41.686722 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:41.582341 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-32.ec2.internal\" not found" Jul 20 21:02:41.686722 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:41.682840 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-32.ec2.internal\" not found" Jul 20 21:02:41.864101 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:41.704262 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-32.ec2.internal" Jul 20 21:02:41.864101 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:41.708109 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-32.ec2.internal" Jul 20 21:02:41.864101 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:41.783892 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-32.ec2.internal\" not found" Jul 20 21:02:41.956263 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:41.884302 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-32.ec2.internal\" not found" Jul 20 21:02:42.031671 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:41.984611 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-32.ec2.internal\" not found" Jul 20 21:02:42.031671 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:42.022895 2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 20 21:02:42.031671 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:42.023027 2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 20 21:02:42.031671 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:42.023038 2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 20 21:02:42.124128 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:42.041661 2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 21:02:42.124128 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:42.084173 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-19 20:57:41 +0000 UTC" deadline="2028-01-24 00:06:11.945973341 +0000 UTC" Jul 20 21:02:42.124128 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:42.084215 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13251h3m29.86176273s" Jul 20 21:02:42.124128 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:42.084763 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-32.ec2.internal\" not found" Jul 20 21:02:42.228620 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:42.185096 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-32.ec2.internal\" not found" Jul 20 21:02:42.319986 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:42.285647 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-32.ec2.internal\" not found" Jul 20 21:02:42.428000 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:42.326512 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ba459730c951195c7a8ffe9d5d50c7d.slice/crio-e5b92a7d379088540c616d1707ccb0e555739c65dd2fcf9638548c685349f0ac WatchSource:0}: Error finding container e5b92a7d379088540c616d1707ccb0e555739c65dd2fcf9638548c685349f0ac: Status 404 returned error can't find the container with id e5b92a7d379088540c616d1707ccb0e555739c65dd2fcf9638548c685349f0ac Jul 20 21:02:42.428000 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:42.326903 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod197924819a9b1fbec1e5ecce03cf3186.slice/crio-7759f7ccda9330d0a47d8e2ab522e1c745782dfe98b38e96a6040bfb55238df5 WatchSource:0}: Error finding container 7759f7ccda9330d0a47d8e2ab522e1c745782dfe98b38e96a6040bfb55238df5: Status 404 returned error can't find the container with id 7759f7ccda9330d0a47d8e2ab522e1c745782dfe98b38e96a6040bfb55238df5 Jul 20 21:02:42.428000 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:42.328378 2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 21:02:42.428000 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:42.331558 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 21:02:42.428000 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:42.383661 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-32.ec2.internal" Jul 20 21:02:42.428000 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:42.395976 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 20 21:02:42.428000 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:42.397005 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-32.ec2.internal" Jul 20 21:02:42.428000 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:42.405624 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 20 21:02:43.062295 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.062252 2562 apiserver.go:52] "Watching apiserver" Jul 20 21:02:43.073319 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.073294 2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 20 21:02:43.073824 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.073800 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-qcsfr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-32.ec2.internal","openshift-multus/multus-additional-cni-plugins-mdg4m","openshift-multus/multus-vz8z6","openshift-network-operator/iptables-alerter-55bk8","openshift-ovn-kubernetes/ovnkube-node-9x7cd","kube-system/global-pull-secret-syncer-2lqmn","kube-system/konnectivity-agent-5jlkf","openshift-cluster-node-tuning-operator/tuned-7z98q","openshift-image-registry/node-ca-lgq8n","openshift-multus/network-metrics-daemon-t79t8","openshift-network-diagnostics/network-check-target-5chx7","kube-system/kube-apiserver-proxy-ip-10-0-136-32.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdpbc"] Jul 20 21:02:43.075857 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.075832 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2lqmn" Jul 20 21:02:43.075968 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:43.075911 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2lqmn" podUID="2b3600ee-a2f5-4b18-ac22-9c868136ce69" Jul 20 21:02:43.077742 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.077716 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t79t8" Jul 20 21:02:43.077878 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:43.077794 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t79t8" podUID="0f09f6b9-6ed0-4165-bc40-706eb42f8f0d" Jul 20 21:02:43.079032 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.079011 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mdg4m" Jul 20 21:02:43.081398 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.081356 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 20 21:02:43.081528 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.081439 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 20 21:02:43.081528 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.081481 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 20 21:02:43.081635 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.081359 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vz8z6" Jul 20 21:02:43.082756 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.082737 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-f45bk\"" Jul 20 21:02:43.082947 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.082934 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 20 21:02:43.084144 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.083739 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 20 21:02:43.084838 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.084266 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 20 21:02:43.085049 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.084964 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 20 21:02:43.085815 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.085796 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-55bk8" Jul 20 21:02:43.086065 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.086047 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-2jzwn\"" Jul 20 21:02:43.087533 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.087511 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.087987 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.087966 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 20 21:02:43.088465 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.088447 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 20 21:02:43.088563 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.088525 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 20 21:02:43.088825 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.088798 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mh44d\"" Jul 20 21:02:43.089102 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.089084 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7z98q" Jul 20 21:02:43.090710 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.090305 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 20 21:02:43.090710 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.090521 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 20 21:02:43.090843 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.090744 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdpbc" Jul 20 21:02:43.092030 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.090898 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 20 21:02:43.092030 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.091111 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 20 21:02:43.092030 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.091172 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wxbll\"" Jul 20 21:02:43.092030 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.091420 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 20 21:02:43.092030 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.091507 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 20 21:02:43.092030 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.091650 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-z22ms\"" Jul 20 21:02:43.092030 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.091775 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 20 21:02:43.092030 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.091885 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 20 21:02:43.092757 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.092310 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5jlkf" Jul 20 21:02:43.092988 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.092967 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 20 21:02:43.094025 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.093852 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 20 21:02:43.094025 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.093885 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 20 21:02:43.094025 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.094019 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wkfpt\"" Jul 20 21:02:43.094792 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.094774 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 20 21:02:43.094792 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.094785 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 20 21:02:43.095027 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.094999 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a853a35b-650d-4ea4-81d3-d4c2ce54e9f1-multus-cni-dir\") pod \"multus-vz8z6\" (UID: \"a853a35b-650d-4ea4-81d3-d4c2ce54e9f1\") " pod="openshift-multus/multus-vz8z6" Jul 20 21:02:43.095081 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.095035 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a853a35b-650d-4ea4-81d3-d4c2ce54e9f1-multus-daemon-config\") pod \"multus-vz8z6\" (UID: \"a853a35b-650d-4ea4-81d3-d4c2ce54e9f1\") " pod="openshift-multus/multus-vz8z6" Jul 20 21:02:43.095081 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.095059 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj457\" (UniqueName: \"kubernetes.io/projected/3d2247cc-c3ac-4a33-a206-a40a152d29fe-kube-api-access-vj457\") pod \"iptables-alerter-55bk8\" (UID: \"3d2247cc-c3ac-4a33-a206-a40a152d29fe\") " pod="openshift-network-operator/iptables-alerter-55bk8" Jul 20 21:02:43.095198 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.095083 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b3600ee-a2f5-4b18-ac22-9c868136ce69-original-pull-secret\") pod \"global-pull-secret-syncer-2lqmn\" (UID: \"2b3600ee-a2f5-4b18-ac22-9c868136ce69\") " pod="kube-system/global-pull-secret-syncer-2lqmn" Jul 20 21:02:43.095198 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.095107 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c4188589-18dc-4bd1-9267-688596e53a7d-cni-binary-copy\") pod \"multus-additional-cni-plugins-mdg4m\" (UID: \"c4188589-18dc-4bd1-9267-688596e53a7d\") " pod="openshift-multus/multus-additional-cni-plugins-mdg4m" Jul 20 21:02:43.095198 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.095145 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c4188589-18dc-4bd1-9267-688596e53a7d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mdg4m\" (UID: \"c4188589-18dc-4bd1-9267-688596e53a7d\") " pod="openshift-multus/multus-additional-cni-plugins-mdg4m" Jul 20 21:02:43.095198 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.095171 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c4188589-18dc-4bd1-9267-688596e53a7d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mdg4m\" (UID: \"c4188589-18dc-4bd1-9267-688596e53a7d\") " pod="openshift-multus/multus-additional-cni-plugins-mdg4m" Jul 20 21:02:43.095404 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.095197 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a853a35b-650d-4ea4-81d3-d4c2ce54e9f1-multus-socket-dir-parent\") pod \"multus-vz8z6\" (UID: \"a853a35b-650d-4ea4-81d3-d4c2ce54e9f1\") " pod="openshift-multus/multus-vz8z6" Jul 20 21:02:43.095404 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.095218 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a853a35b-650d-4ea4-81d3-d4c2ce54e9f1-multus-conf-dir\") pod \"multus-vz8z6\" (UID: \"a853a35b-650d-4ea4-81d3-d4c2ce54e9f1\") " pod="openshift-multus/multus-vz8z6" Jul 20 21:02:43.095404 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.095228 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-55fdk\"" Jul 20 21:02:43.095404 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.095244 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c4188589-18dc-4bd1-9267-688596e53a7d-os-release\") pod \"multus-additional-cni-plugins-mdg4m\" (UID: \"c4188589-18dc-4bd1-9267-688596e53a7d\") " pod="openshift-multus/multus-additional-cni-plugins-mdg4m" Jul 20 21:02:43.095404 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.095269 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb7d5\" (UniqueName: \"kubernetes.io/projected/c4188589-18dc-4bd1-9267-688596e53a7d-kube-api-access-rb7d5\") pod \"multus-additional-cni-plugins-mdg4m\" (UID: \"c4188589-18dc-4bd1-9267-688596e53a7d\") " pod="openshift-multus/multus-additional-cni-plugins-mdg4m" Jul 20 21:02:43.095404 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.095292 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq5fr\" (UniqueName: \"kubernetes.io/projected/a853a35b-650d-4ea4-81d3-d4c2ce54e9f1-kube-api-access-rq5fr\") pod \"multus-vz8z6\" (UID: \"a853a35b-650d-4ea4-81d3-d4c2ce54e9f1\") " pod="openshift-multus/multus-vz8z6" Jul 20 21:02:43.095404 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.095313 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2b3600ee-a2f5-4b18-ac22-9c868136ce69-dbus\") pod \"global-pull-secret-syncer-2lqmn\" (UID: \"2b3600ee-a2f5-4b18-ac22-9c868136ce69\") " pod="kube-system/global-pull-secret-syncer-2lqmn" Jul 20 21:02:43.095404 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.095335 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a853a35b-650d-4ea4-81d3-d4c2ce54e9f1-system-cni-dir\") pod \"multus-vz8z6\" (UID: \"a853a35b-650d-4ea4-81d3-d4c2ce54e9f1\") " pod="openshift-multus/multus-vz8z6" Jul 20 21:02:43.095404 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.095355 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a853a35b-650d-4ea4-81d3-d4c2ce54e9f1-cnibin\") pod \"multus-vz8z6\" (UID: \"a853a35b-650d-4ea4-81d3-d4c2ce54e9f1\") " pod="openshift-multus/multus-vz8z6" Jul 20 21:02:43.095404 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.095375 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a853a35b-650d-4ea4-81d3-d4c2ce54e9f1-cni-binary-copy\") pod \"multus-vz8z6\" (UID: \"a853a35b-650d-4ea4-81d3-d4c2ce54e9f1\") " pod="openshift-multus/multus-vz8z6" Jul 20 21:02:43.095404 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.095400 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a853a35b-650d-4ea4-81d3-d4c2ce54e9f1-host-var-lib-cni-multus\") pod \"multus-vz8z6\" (UID: \"a853a35b-650d-4ea4-81d3-d4c2ce54e9f1\") " pod="openshift-multus/multus-vz8z6" Jul 20 21:02:43.095907 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.095421 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a853a35b-650d-4ea4-81d3-d4c2ce54e9f1-host-run-multus-certs\") pod \"multus-vz8z6\" (UID: \"a853a35b-650d-4ea4-81d3-d4c2ce54e9f1\") " pod="openshift-multus/multus-vz8z6" Jul 20 21:02:43.095907 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.095453 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3d2247cc-c3ac-4a33-a206-a40a152d29fe-host-slash\") pod \"iptables-alerter-55bk8\" (UID: \"3d2247cc-c3ac-4a33-a206-a40a152d29fe\") " pod="openshift-network-operator/iptables-alerter-55bk8" Jul 20 21:02:43.095907 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.095487 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a853a35b-650d-4ea4-81d3-d4c2ce54e9f1-os-release\") pod \"multus-vz8z6\" (UID: \"a853a35b-650d-4ea4-81d3-d4c2ce54e9f1\") " pod="openshift-multus/multus-vz8z6" Jul 20 21:02:43.095907 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.095511 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a853a35b-650d-4ea4-81d3-d4c2ce54e9f1-host-run-netns\") pod \"multus-vz8z6\" (UID: \"a853a35b-650d-4ea4-81d3-d4c2ce54e9f1\") " pod="openshift-multus/multus-vz8z6" Jul 20 21:02:43.095907 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.095565 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f09f6b9-6ed0-4165-bc40-706eb42f8f0d-metrics-certs\") pod \"network-metrics-daemon-t79t8\" (UID: \"0f09f6b9-6ed0-4165-bc40-706eb42f8f0d\") " pod="openshift-multus/network-metrics-daemon-t79t8" Jul 20 21:02:43.095907 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.095590 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mpps\" (UniqueName: \"kubernetes.io/projected/0f09f6b9-6ed0-4165-bc40-706eb42f8f0d-kube-api-access-6mpps\") pod \"network-metrics-daemon-t79t8\" (UID: \"0f09f6b9-6ed0-4165-bc40-706eb42f8f0d\") " pod="openshift-multus/network-metrics-daemon-t79t8" Jul 20 21:02:43.095907 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.095616 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a853a35b-650d-4ea4-81d3-d4c2ce54e9f1-host-run-k8s-cni-cncf-io\") pod \"multus-vz8z6\" (UID: \"a853a35b-650d-4ea4-81d3-d4c2ce54e9f1\") " pod="openshift-multus/multus-vz8z6" Jul 20 21:02:43.095907 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.095638 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3d2247cc-c3ac-4a33-a206-a40a152d29fe-iptables-alerter-script\") pod \"iptables-alerter-55bk8\" (UID: \"3d2247cc-c3ac-4a33-a206-a40a152d29fe\") " pod="openshift-network-operator/iptables-alerter-55bk8" Jul 20 21:02:43.095907 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.095664 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4188589-18dc-4bd1-9267-688596e53a7d-system-cni-dir\") pod \"multus-additional-cni-plugins-mdg4m\" (UID: \"c4188589-18dc-4bd1-9267-688596e53a7d\") " pod="openshift-multus/multus-additional-cni-plugins-mdg4m" Jul 20 21:02:43.095907 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.095687 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a853a35b-650d-4ea4-81d3-d4c2ce54e9f1-host-var-lib-kubelet\") pod \"multus-vz8z6\" (UID: \"a853a35b-650d-4ea4-81d3-d4c2ce54e9f1\") " pod="openshift-multus/multus-vz8z6" Jul 20 21:02:43.095907 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.095709 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a853a35b-650d-4ea4-81d3-d4c2ce54e9f1-etc-kubernetes\") pod \"multus-vz8z6\" (UID: \"a853a35b-650d-4ea4-81d3-d4c2ce54e9f1\") " pod="openshift-multus/multus-vz8z6" Jul 20 21:02:43.095907 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.095733 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2b3600ee-a2f5-4b18-ac22-9c868136ce69-kubelet-config\") pod \"global-pull-secret-syncer-2lqmn\" (UID: \"2b3600ee-a2f5-4b18-ac22-9c868136ce69\") " pod="kube-system/global-pull-secret-syncer-2lqmn" Jul 20 21:02:43.095907 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.095756 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a853a35b-650d-4ea4-81d3-d4c2ce54e9f1-host-var-lib-cni-bin\") pod \"multus-vz8z6\" (UID: \"a853a35b-650d-4ea4-81d3-d4c2ce54e9f1\") " pod="openshift-multus/multus-vz8z6" Jul 20 21:02:43.095907 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.095799 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a853a35b-650d-4ea4-81d3-d4c2ce54e9f1-hostroot\") pod \"multus-vz8z6\" (UID: \"a853a35b-650d-4ea4-81d3-d4c2ce54e9f1\") " pod="openshift-multus/multus-vz8z6" Jul 20 21:02:43.096674 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.095831 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c4188589-18dc-4bd1-9267-688596e53a7d-cnibin\") pod \"multus-additional-cni-plugins-mdg4m\" (UID: \"c4188589-18dc-4bd1-9267-688596e53a7d\") " pod="openshift-multus/multus-additional-cni-plugins-mdg4m" Jul 20 21:02:43.096674 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.095854 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c4188589-18dc-4bd1-9267-688596e53a7d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mdg4m\" (UID: \"c4188589-18dc-4bd1-9267-688596e53a7d\") " pod="openshift-multus/multus-additional-cni-plugins-mdg4m" Jul 20 21:02:43.096674 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.095970 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lgq8n" Jul 20 21:02:43.097932 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.097518 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qcsfr" Jul 20 21:02:43.098939 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.098703 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 20 21:02:43.099724 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.098975 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 20 21:02:43.099724 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.099189 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bcnwv\"" Jul 20 21:02:43.099724 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.099432 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5chx7" Jul 20 21:02:43.099724 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.099483 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 20 21:02:43.099724 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:43.099493 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5chx7" podUID="f0fef424-8965-40f7-8b13-5a72773f050a" Jul 20 21:02:43.099952 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.099808 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-q44wf\"" Jul 20 21:02:43.100537 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.100515 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 20 21:02:43.100634 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.100615 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 20 21:02:43.101210 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.101194 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 20 21:02:43.139052 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.138855 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-mtls7" Jul 20 21:02:43.149767 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.149745 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mtls7" Jul 20 21:02:43.185710 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.185680 2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 20 21:02:43.196656 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.196626 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a853a35b-650d-4ea4-81d3-d4c2ce54e9f1-os-release\") pod \"multus-vz8z6\" (UID: \"a853a35b-650d-4ea4-81d3-d4c2ce54e9f1\") " pod="openshift-multus/multus-vz8z6" Jul 20 21:02:43.196800 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.196666 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/356480cb-3559-4be3-a882-45d97dd6e493-host-run-netns\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.196800 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.196694 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a853a35b-650d-4ea4-81d3-d4c2ce54e9f1-host-run-netns\") pod \"multus-vz8z6\" (UID: \"a853a35b-650d-4ea4-81d3-d4c2ce54e9f1\") " pod="openshift-multus/multus-vz8z6" Jul 20 21:02:43.196800 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.196719 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f09f6b9-6ed0-4165-bc40-706eb42f8f0d-metrics-certs\") pod \"network-metrics-daemon-t79t8\" (UID: \"0f09f6b9-6ed0-4165-bc40-706eb42f8f0d\") " pod="openshift-multus/network-metrics-daemon-t79t8" Jul 20 21:02:43.196800 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.196755 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3d2247cc-c3ac-4a33-a206-a40a152d29fe-iptables-alerter-script\") pod \"iptables-alerter-55bk8\" (UID: \"3d2247cc-c3ac-4a33-a206-a40a152d29fe\") " pod="openshift-network-operator/iptables-alerter-55bk8" Jul 20 21:02:43.196800 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.196776 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a853a35b-650d-4ea4-81d3-d4c2ce54e9f1-os-release\") pod \"multus-vz8z6\" (UID: \"a853a35b-650d-4ea4-81d3-d4c2ce54e9f1\") " pod="openshift-multus/multus-vz8z6" Jul 20 21:02:43.197073 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.196780 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gb6r\" (UniqueName: \"kubernetes.io/projected/68cad356-4b30-41ba-b68f-281c78eb24c1-kube-api-access-5gb6r\") pod \"aws-ebs-csi-driver-node-vdpbc\" (UID: \"68cad356-4b30-41ba-b68f-281c78eb24c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdpbc" Jul 20 21:02:43.197073 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.196850 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a853a35b-650d-4ea4-81d3-d4c2ce54e9f1-host-run-netns\") pod \"multus-vz8z6\" (UID: \"a853a35b-650d-4ea4-81d3-d4c2ce54e9f1\") " pod="openshift-multus/multus-vz8z6" Jul 20 21:02:43.197073 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.196890 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ef68231f-0506-4450-9fcb-bc14af3b7d76-serviceca\") pod \"node-ca-lgq8n\" (UID: \"ef68231f-0506-4450-9fcb-bc14af3b7d76\") " pod="openshift-image-registry/node-ca-lgq8n" Jul 20 21:02:43.197073 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.196913 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/356480cb-3559-4be3-a882-45d97dd6e493-run-ovn\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.197073 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.196943 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/356480cb-3559-4be3-a882-45d97dd6e493-ovnkube-script-lib\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.197073 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.196967 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e253d85e-aa94-494c-aaaf-daa7d9490119-etc-sysconfig\") pod \"tuned-7z98q\" (UID: \"e253d85e-aa94-494c-aaaf-daa7d9490119\") " pod="openshift-cluster-node-tuning-operator/tuned-7z98q" Jul 20 21:02:43.197073 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:43.197024 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 21:02:43.197430 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.197078 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/356480cb-3559-4be3-a882-45d97dd6e493-var-lib-openvswitch\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.197430 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:43.197117 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f09f6b9-6ed0-4165-bc40-706eb42f8f0d-metrics-certs podName:0f09f6b9-6ed0-4165-bc40-706eb42f8f0d nodeName:}" failed. No retries permitted until 2026-07-20 21:02:43.697079977 +0000 UTC m=+3.022782911 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0f09f6b9-6ed0-4165-bc40-706eb42f8f0d-metrics-certs") pod "network-metrics-daemon-t79t8" (UID: "0f09f6b9-6ed0-4165-bc40-706eb42f8f0d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 21:02:43.197430 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.197158 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/356480cb-3559-4be3-a882-45d97dd6e493-etc-openvswitch\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.197430 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.197181 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/356480cb-3559-4be3-a882-45d97dd6e493-host-cni-netd\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.197430 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.197206 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a853a35b-650d-4ea4-81d3-d4c2ce54e9f1-host-var-lib-cni-bin\") pod \"multus-vz8z6\" (UID: \"a853a35b-650d-4ea4-81d3-d4c2ce54e9f1\") " pod="openshift-multus/multus-vz8z6" Jul 20 21:02:43.197430 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.197228 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a853a35b-650d-4ea4-81d3-d4c2ce54e9f1-hostroot\") pod \"multus-vz8z6\" (UID: \"a853a35b-650d-4ea4-81d3-d4c2ce54e9f1\") " pod="openshift-multus/multus-vz8z6" Jul 20 21:02:43.197430 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.197250 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/68cad356-4b30-41ba-b68f-281c78eb24c1-registration-dir\") pod \"aws-ebs-csi-driver-node-vdpbc\" (UID: \"68cad356-4b30-41ba-b68f-281c78eb24c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdpbc" Jul 20 21:02:43.197430 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.197273 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/971c4fcc-cdb1-460e-9d0b-6eb07367f5c6-konnectivity-ca\") pod \"konnectivity-agent-5jlkf\" (UID: \"971c4fcc-cdb1-460e-9d0b-6eb07367f5c6\") " pod="kube-system/konnectivity-agent-5jlkf" Jul 20 21:02:43.197430 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.197303 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c4188589-18dc-4bd1-9267-688596e53a7d-cnibin\") pod \"multus-additional-cni-plugins-mdg4m\" (UID: \"c4188589-18dc-4bd1-9267-688596e53a7d\") " pod="openshift-multus/multus-additional-cni-plugins-mdg4m" Jul 20 21:02:43.197430 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.197326 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e253d85e-aa94-494c-aaaf-daa7d9490119-etc-sysctl-conf\") pod \"tuned-7z98q\" (UID: \"e253d85e-aa94-494c-aaaf-daa7d9490119\") " pod="openshift-cluster-node-tuning-operator/tuned-7z98q" Jul 20 21:02:43.197430 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.197349 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e253d85e-aa94-494c-aaaf-daa7d9490119-var-lib-kubelet\") pod \"tuned-7z98q\" (UID: \"e253d85e-aa94-494c-aaaf-daa7d9490119\") " pod="openshift-cluster-node-tuning-operator/tuned-7z98q" Jul 20 21:02:43.197430 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.197352 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a853a35b-650d-4ea4-81d3-d4c2ce54e9f1-host-var-lib-cni-bin\") pod \"multus-vz8z6\" (UID: \"a853a35b-650d-4ea4-81d3-d4c2ce54e9f1\") " pod="openshift-multus/multus-vz8z6" Jul 20 21:02:43.197430 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.197374 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzz2n\" (UniqueName: \"kubernetes.io/projected/f0fef424-8965-40f7-8b13-5a72773f050a-kube-api-access-hzz2n\") pod \"network-check-target-5chx7\" (UID: \"f0fef424-8965-40f7-8b13-5a72773f050a\") " pod="openshift-network-diagnostics/network-check-target-5chx7" Jul 20 21:02:43.198041 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.197400 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c4188589-18dc-4bd1-9267-688596e53a7d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mdg4m\" (UID: \"c4188589-18dc-4bd1-9267-688596e53a7d\") " pod="openshift-multus/multus-additional-cni-plugins-mdg4m" Jul 20 21:02:43.198041 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.197406 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3d2247cc-c3ac-4a33-a206-a40a152d29fe-iptables-alerter-script\") pod \"iptables-alerter-55bk8\" (UID: \"3d2247cc-c3ac-4a33-a206-a40a152d29fe\") " pod="openshift-network-operator/iptables-alerter-55bk8" Jul 20 21:02:43.198041 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.197450 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a853a35b-650d-4ea4-81d3-d4c2ce54e9f1-hostroot\") pod \"multus-vz8z6\" (UID: \"a853a35b-650d-4ea4-81d3-d4c2ce54e9f1\") " pod="openshift-multus/multus-vz8z6" Jul 20 21:02:43.198041 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.197449 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c4188589-18dc-4bd1-9267-688596e53a7d-cnibin\") pod \"multus-additional-cni-plugins-mdg4m\" (UID: \"c4188589-18dc-4bd1-9267-688596e53a7d\") " pod="openshift-multus/multus-additional-cni-plugins-mdg4m" Jul 20 21:02:43.198041 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.197467 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a853a35b-650d-4ea4-81d3-d4c2ce54e9f1-multus-socket-dir-parent\") pod \"multus-vz8z6\" (UID: \"a853a35b-650d-4ea4-81d3-d4c2ce54e9f1\") " pod="openshift-multus/multus-vz8z6" Jul 20 21:02:43.198041 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.197492 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/356480cb-3559-4be3-a882-45d97dd6e493-host-cni-bin\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.198041 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.197515 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/356480cb-3559-4be3-a882-45d97dd6e493-env-overrides\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.198041 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.197534 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb7d5\" (UniqueName: \"kubernetes.io/projected/c4188589-18dc-4bd1-9267-688596e53a7d-kube-api-access-rb7d5\") pod \"multus-additional-cni-plugins-mdg4m\" (UID: \"c4188589-18dc-4bd1-9267-688596e53a7d\") " pod="openshift-multus/multus-additional-cni-plugins-mdg4m" Jul 20 21:02:43.198041 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.197571 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/356480cb-3559-4be3-a882-45d97dd6e493-run-systemd\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.198041 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.197629 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a853a35b-650d-4ea4-81d3-d4c2ce54e9f1-multus-socket-dir-parent\") pod \"multus-vz8z6\" (UID: \"a853a35b-650d-4ea4-81d3-d4c2ce54e9f1\") " pod="openshift-multus/multus-vz8z6" Jul 20 21:02:43.198041 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.197653 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/356480cb-3559-4be3-a882-45d97dd6e493-log-socket\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.198041 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.197689 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/356480cb-3559-4be3-a882-45d97dd6e493-ovn-node-metrics-cert\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.198041 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.197715 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e253d85e-aa94-494c-aaaf-daa7d9490119-lib-modules\") pod \"tuned-7z98q\" (UID: \"e253d85e-aa94-494c-aaaf-daa7d9490119\") " pod="openshift-cluster-node-tuning-operator/tuned-7z98q" Jul 20 21:02:43.198041 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.197755 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/971c4fcc-cdb1-460e-9d0b-6eb07367f5c6-agent-certs\") pod \"konnectivity-agent-5jlkf\" (UID: \"971c4fcc-cdb1-460e-9d0b-6eb07367f5c6\") " pod="kube-system/konnectivity-agent-5jlkf" Jul 20 21:02:43.198684 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.197779 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2b3600ee-a2f5-4b18-ac22-9c868136ce69-dbus\") pod \"global-pull-secret-syncer-2lqmn\" (UID: \"2b3600ee-a2f5-4b18-ac22-9c868136ce69\") " pod="kube-system/global-pull-secret-syncer-2lqmn" Jul 20 21:02:43.198684 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.197806 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a853a35b-650d-4ea4-81d3-d4c2ce54e9f1-system-cni-dir\") pod \"multus-vz8z6\" (UID: \"a853a35b-650d-4ea4-81d3-d4c2ce54e9f1\") " pod="openshift-multus/multus-vz8z6" Jul 20 21:02:43.198684 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.197831 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a853a35b-650d-4ea4-81d3-d4c2ce54e9f1-cni-binary-copy\") pod \"multus-vz8z6\" (UID: \"a853a35b-650d-4ea4-81d3-d4c2ce54e9f1\") " pod="openshift-multus/multus-vz8z6" Jul 20 21:02:43.198684 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.197853 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a853a35b-650d-4ea4-81d3-d4c2ce54e9f1-host-var-lib-cni-multus\") pod \"multus-vz8z6\" (UID: \"a853a35b-650d-4ea4-81d3-d4c2ce54e9f1\") " pod="openshift-multus/multus-vz8z6" Jul 20 21:02:43.198684 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.197868 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a853a35b-650d-4ea4-81d3-d4c2ce54e9f1-system-cni-dir\") pod \"multus-vz8z6\" (UID: \"a853a35b-650d-4ea4-81d3-d4c2ce54e9f1\") " pod="openshift-multus/multus-vz8z6" Jul 20 21:02:43.198684 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.197892 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a853a35b-650d-4ea4-81d3-d4c2ce54e9f1-host-run-multus-certs\") pod \"multus-vz8z6\" (UID: \"a853a35b-650d-4ea4-81d3-d4c2ce54e9f1\") " pod="openshift-multus/multus-vz8z6" Jul 20 21:02:43.198684 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.197893 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2b3600ee-a2f5-4b18-ac22-9c868136ce69-dbus\") pod \"global-pull-secret-syncer-2lqmn\" (UID: \"2b3600ee-a2f5-4b18-ac22-9c868136ce69\") " pod="kube-system/global-pull-secret-syncer-2lqmn" Jul 20 21:02:43.198684 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.197917 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/68cad356-4b30-41ba-b68f-281c78eb24c1-etc-selinux\") pod \"aws-ebs-csi-driver-node-vdpbc\" (UID: \"68cad356-4b30-41ba-b68f-281c78eb24c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdpbc" Jul 20 21:02:43.198684 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.197925 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c4188589-18dc-4bd1-9267-688596e53a7d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mdg4m\" (UID: \"c4188589-18dc-4bd1-9267-688596e53a7d\") " pod="openshift-multus/multus-additional-cni-plugins-mdg4m" Jul 20 21:02:43.198684 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.197938 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a853a35b-650d-4ea4-81d3-d4c2ce54e9f1-host-var-lib-cni-multus\") pod \"multus-vz8z6\" (UID: \"a853a35b-650d-4ea4-81d3-d4c2ce54e9f1\") " pod="openshift-multus/multus-vz8z6" Jul 20 21:02:43.198684 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.197940 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/356480cb-3559-4be3-a882-45d97dd6e493-ovnkube-config\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.198684 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.197975 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e253d85e-aa94-494c-aaaf-daa7d9490119-etc-systemd\") pod \"tuned-7z98q\" (UID: \"e253d85e-aa94-494c-aaaf-daa7d9490119\") " pod="openshift-cluster-node-tuning-operator/tuned-7z98q" Jul 20 21:02:43.198684 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.197974 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a853a35b-650d-4ea4-81d3-d4c2ce54e9f1-host-run-multus-certs\") pod \"multus-vz8z6\" (UID: \"a853a35b-650d-4ea4-81d3-d4c2ce54e9f1\") " pod="openshift-multus/multus-vz8z6" Jul 20 21:02:43.198684 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.198006 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f8gv\" (UniqueName: \"kubernetes.io/projected/ef68231f-0506-4450-9fcb-bc14af3b7d76-kube-api-access-6f8gv\") pod \"node-ca-lgq8n\" (UID: \"ef68231f-0506-4450-9fcb-bc14af3b7d76\") " pod="openshift-image-registry/node-ca-lgq8n" Jul 20 21:02:43.199370 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.198030 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/356480cb-3559-4be3-a882-45d97dd6e493-run-openvswitch\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.199370 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.198043 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e253d85e-aa94-494c-aaaf-daa7d9490119-run\") pod \"tuned-7z98q\" (UID: \"e253d85e-aa94-494c-aaaf-daa7d9490119\") " pod="openshift-cluster-node-tuning-operator/tuned-7z98q" Jul 20 21:02:43.199370 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.198085 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e253d85e-aa94-494c-aaaf-daa7d9490119-sys\") pod \"tuned-7z98q\" (UID: \"e253d85e-aa94-494c-aaaf-daa7d9490119\") " pod="openshift-cluster-node-tuning-operator/tuned-7z98q" Jul 20 21:02:43.199370 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.198113 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e253d85e-aa94-494c-aaaf-daa7d9490119-tmp\") pod \"tuned-7z98q\" (UID: \"e253d85e-aa94-494c-aaaf-daa7d9490119\") " pod="openshift-cluster-node-tuning-operator/tuned-7z98q" Jul 20 21:02:43.199370 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.198149 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dde8ddcb-7c86-4c9d-bd00-11129c8ca35e-hosts-file\") pod \"node-resolver-qcsfr\" (UID: \"dde8ddcb-7c86-4c9d-bd00-11129c8ca35e\") " pod="openshift-dns/node-resolver-qcsfr" Jul 20 21:02:43.199370 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.198180 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq7k9\" (UniqueName: \"kubernetes.io/projected/dde8ddcb-7c86-4c9d-bd00-11129c8ca35e-kube-api-access-pq7k9\") pod \"node-resolver-qcsfr\" (UID: \"dde8ddcb-7c86-4c9d-bd00-11129c8ca35e\") " pod="openshift-dns/node-resolver-qcsfr" Jul 20 21:02:43.199370 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.198209 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mpps\" (UniqueName: \"kubernetes.io/projected/0f09f6b9-6ed0-4165-bc40-706eb42f8f0d-kube-api-access-6mpps\") pod \"network-metrics-daemon-t79t8\" (UID: \"0f09f6b9-6ed0-4165-bc40-706eb42f8f0d\") " pod="openshift-multus/network-metrics-daemon-t79t8" Jul 20 21:02:43.199370 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.198232 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a853a35b-650d-4ea4-81d3-d4c2ce54e9f1-host-run-k8s-cni-cncf-io\") pod \"multus-vz8z6\" (UID: \"a853a35b-650d-4ea4-81d3-d4c2ce54e9f1\") " pod="openshift-multus/multus-vz8z6" Jul 20 21:02:43.199370 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.198275 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a853a35b-650d-4ea4-81d3-d4c2ce54e9f1-host-run-k8s-cni-cncf-io\") pod \"multus-vz8z6\" (UID: \"a853a35b-650d-4ea4-81d3-d4c2ce54e9f1\") " pod="openshift-multus/multus-vz8z6" Jul 20 21:02:43.199370 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.198321 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a853a35b-650d-4ea4-81d3-d4c2ce54e9f1-cni-binary-copy\") pod \"multus-vz8z6\" (UID: \"a853a35b-650d-4ea4-81d3-d4c2ce54e9f1\") " pod="openshift-multus/multus-vz8z6" Jul 20 21:02:43.199370 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.198356 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/68cad356-4b30-41ba-b68f-281c78eb24c1-device-dir\") pod \"aws-ebs-csi-driver-node-vdpbc\" (UID: \"68cad356-4b30-41ba-b68f-281c78eb24c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdpbc" Jul 20 21:02:43.199370 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.198386 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/356480cb-3559-4be3-a882-45d97dd6e493-host-run-ovn-kubernetes\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.199370 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.198408 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e253d85e-aa94-494c-aaaf-daa7d9490119-etc-kubernetes\") pod \"tuned-7z98q\" (UID: \"e253d85e-aa94-494c-aaaf-daa7d9490119\") " pod="openshift-cluster-node-tuning-operator/tuned-7z98q" Jul 20 21:02:43.199370 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.198439 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4188589-18dc-4bd1-9267-688596e53a7d-system-cni-dir\") pod \"multus-additional-cni-plugins-mdg4m\" (UID: \"c4188589-18dc-4bd1-9267-688596e53a7d\") " pod="openshift-multus/multus-additional-cni-plugins-mdg4m" Jul 20 21:02:43.200059 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.198468 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4188589-18dc-4bd1-9267-688596e53a7d-system-cni-dir\") pod \"multus-additional-cni-plugins-mdg4m\" (UID: \"c4188589-18dc-4bd1-9267-688596e53a7d\") " pod="openshift-multus/multus-additional-cni-plugins-mdg4m" Jul 20 21:02:43.200059 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.198490 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a853a35b-650d-4ea4-81d3-d4c2ce54e9f1-host-var-lib-kubelet\") pod \"multus-vz8z6\" (UID: \"a853a35b-650d-4ea4-81d3-d4c2ce54e9f1\") " pod="openshift-multus/multus-vz8z6" Jul 20 21:02:43.200059 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.198511 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a853a35b-650d-4ea4-81d3-d4c2ce54e9f1-etc-kubernetes\") pod \"multus-vz8z6\" (UID: \"a853a35b-650d-4ea4-81d3-d4c2ce54e9f1\") " pod="openshift-multus/multus-vz8z6" Jul 20 21:02:43.200059 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.198528 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/356480cb-3559-4be3-a882-45d97dd6e493-node-log\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.200059 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.198541 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a853a35b-650d-4ea4-81d3-d4c2ce54e9f1-host-var-lib-kubelet\") pod \"multus-vz8z6\" (UID: \"a853a35b-650d-4ea4-81d3-d4c2ce54e9f1\") " pod="openshift-multus/multus-vz8z6" Jul 20 21:02:43.200059 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.198544 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2b3600ee-a2f5-4b18-ac22-9c868136ce69-kubelet-config\") pod \"global-pull-secret-syncer-2lqmn\" (UID: \"2b3600ee-a2f5-4b18-ac22-9c868136ce69\") " pod="kube-system/global-pull-secret-syncer-2lqmn" Jul 20 21:02:43.200059 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.198578 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2b3600ee-a2f5-4b18-ac22-9c868136ce69-kubelet-config\") pod \"global-pull-secret-syncer-2lqmn\" (UID: \"2b3600ee-a2f5-4b18-ac22-9c868136ce69\") " pod="kube-system/global-pull-secret-syncer-2lqmn" Jul 20 21:02:43.200059 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.198591 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/356480cb-3559-4be3-a882-45d97dd6e493-host-kubelet\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.200059 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.198614 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a853a35b-650d-4ea4-81d3-d4c2ce54e9f1-etc-kubernetes\") pod \"multus-vz8z6\" (UID: \"a853a35b-650d-4ea4-81d3-d4c2ce54e9f1\") " pod="openshift-multus/multus-vz8z6" Jul 20 21:02:43.200059 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.198624 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e253d85e-aa94-494c-aaaf-daa7d9490119-etc-modprobe-d\") pod \"tuned-7z98q\" (UID: \"e253d85e-aa94-494c-aaaf-daa7d9490119\") " pod="openshift-cluster-node-tuning-operator/tuned-7z98q" Jul 20 21:02:43.200059 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.198661 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c4188589-18dc-4bd1-9267-688596e53a7d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mdg4m\" (UID: \"c4188589-18dc-4bd1-9267-688596e53a7d\") " pod="openshift-multus/multus-additional-cni-plugins-mdg4m" Jul 20 21:02:43.200059 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.198688 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a853a35b-650d-4ea4-81d3-d4c2ce54e9f1-multus-cni-dir\") pod \"multus-vz8z6\" (UID: \"a853a35b-650d-4ea4-81d3-d4c2ce54e9f1\") " pod="openshift-multus/multus-vz8z6" Jul 20 21:02:43.200059 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.198711 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a853a35b-650d-4ea4-81d3-d4c2ce54e9f1-multus-daemon-config\") pod \"multus-vz8z6\" (UID: \"a853a35b-650d-4ea4-81d3-d4c2ce54e9f1\") " pod="openshift-multus/multus-vz8z6" Jul 20 21:02:43.200059 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.198727 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj457\" (UniqueName: \"kubernetes.io/projected/3d2247cc-c3ac-4a33-a206-a40a152d29fe-kube-api-access-vj457\") pod \"iptables-alerter-55bk8\" (UID: \"3d2247cc-c3ac-4a33-a206-a40a152d29fe\") " pod="openshift-network-operator/iptables-alerter-55bk8" Jul 20 21:02:43.200748 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.198750 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/68cad356-4b30-41ba-b68f-281c78eb24c1-socket-dir\") pod \"aws-ebs-csi-driver-node-vdpbc\" (UID: \"68cad356-4b30-41ba-b68f-281c78eb24c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdpbc" Jul 20 21:02:43.200748 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.198822 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a853a35b-650d-4ea4-81d3-d4c2ce54e9f1-multus-cni-dir\") pod \"multus-vz8z6\" (UID: \"a853a35b-650d-4ea4-81d3-d4c2ce54e9f1\") " pod="openshift-multus/multus-vz8z6" Jul 20 21:02:43.200748 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.198861 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dde8ddcb-7c86-4c9d-bd00-11129c8ca35e-tmp-dir\") pod \"node-resolver-qcsfr\" (UID: \"dde8ddcb-7c86-4c9d-bd00-11129c8ca35e\") " pod="openshift-dns/node-resolver-qcsfr" Jul 20 21:02:43.200748 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.198894 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/356480cb-3559-4be3-a882-45d97dd6e493-systemd-units\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.200748 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.198920 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b3600ee-a2f5-4b18-ac22-9c868136ce69-original-pull-secret\") pod \"global-pull-secret-syncer-2lqmn\" (UID: \"2b3600ee-a2f5-4b18-ac22-9c868136ce69\") " pod="kube-system/global-pull-secret-syncer-2lqmn" Jul 20 21:02:43.200748 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.198944 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c4188589-18dc-4bd1-9267-688596e53a7d-cni-binary-copy\") pod \"multus-additional-cni-plugins-mdg4m\" (UID: \"c4188589-18dc-4bd1-9267-688596e53a7d\") " pod="openshift-multus/multus-additional-cni-plugins-mdg4m" Jul 20 21:02:43.200748 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.198967 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c4188589-18dc-4bd1-9267-688596e53a7d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mdg4m\" (UID: \"c4188589-18dc-4bd1-9267-688596e53a7d\") " pod="openshift-multus/multus-additional-cni-plugins-mdg4m" Jul 20 21:02:43.200748 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.198990 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a853a35b-650d-4ea4-81d3-d4c2ce54e9f1-multus-conf-dir\") pod \"multus-vz8z6\" (UID: \"a853a35b-650d-4ea4-81d3-d4c2ce54e9f1\") " pod="openshift-multus/multus-vz8z6" Jul 20 21:02:43.200748 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.199013 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/68cad356-4b30-41ba-b68f-281c78eb24c1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vdpbc\" (UID: \"68cad356-4b30-41ba-b68f-281c78eb24c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdpbc" Jul 20 21:02:43.200748 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.199038 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ef68231f-0506-4450-9fcb-bc14af3b7d76-host\") pod \"node-ca-lgq8n\" (UID: \"ef68231f-0506-4450-9fcb-bc14af3b7d76\") " pod="openshift-image-registry/node-ca-lgq8n" Jul 20 21:02:43.200748 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:43.199076 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 21:02:43.200748 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:43.199125 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b3600ee-a2f5-4b18-ac22-9c868136ce69-original-pull-secret podName:2b3600ee-a2f5-4b18-ac22-9c868136ce69 nodeName:}" failed. No retries permitted until 2026-07-20 21:02:43.699109333 +0000 UTC m=+3.024812257 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b3600ee-a2f5-4b18-ac22-9c868136ce69-original-pull-secret") pod "global-pull-secret-syncer-2lqmn" (UID: "2b3600ee-a2f5-4b18-ac22-9c868136ce69") : object "kube-system"/"original-pull-secret" not registered Jul 20 21:02:43.200748 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.199128 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c4188589-18dc-4bd1-9267-688596e53a7d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mdg4m\" (UID: \"c4188589-18dc-4bd1-9267-688596e53a7d\") " pod="openshift-multus/multus-additional-cni-plugins-mdg4m" Jul 20 21:02:43.200748 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.199076 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctmx4\" (UniqueName: \"kubernetes.io/projected/356480cb-3559-4be3-a882-45d97dd6e493-kube-api-access-ctmx4\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.201475 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.199201 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a853a35b-650d-4ea4-81d3-d4c2ce54e9f1-multus-conf-dir\") pod \"multus-vz8z6\" (UID: \"a853a35b-650d-4ea4-81d3-d4c2ce54e9f1\") " pod="openshift-multus/multus-vz8z6" Jul 20 21:02:43.201475 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.199207 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c4188589-18dc-4bd1-9267-688596e53a7d-os-release\") pod \"multus-additional-cni-plugins-mdg4m\" (UID: \"c4188589-18dc-4bd1-9267-688596e53a7d\") " pod="openshift-multus/multus-additional-cni-plugins-mdg4m" Jul 20 21:02:43.201475 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.199230 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a853a35b-650d-4ea4-81d3-d4c2ce54e9f1-multus-daemon-config\") pod \"multus-vz8z6\" (UID: \"a853a35b-650d-4ea4-81d3-d4c2ce54e9f1\") " pod="openshift-multus/multus-vz8z6" Jul 20 21:02:43.201475 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.199243 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq5fr\" (UniqueName: \"kubernetes.io/projected/a853a35b-650d-4ea4-81d3-d4c2ce54e9f1-kube-api-access-rq5fr\") pod \"multus-vz8z6\" (UID: \"a853a35b-650d-4ea4-81d3-d4c2ce54e9f1\") " pod="openshift-multus/multus-vz8z6" Jul 20 21:02:43.201475 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.199276 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c4188589-18dc-4bd1-9267-688596e53a7d-os-release\") pod \"multus-additional-cni-plugins-mdg4m\" (UID: \"c4188589-18dc-4bd1-9267-688596e53a7d\") " pod="openshift-multus/multus-additional-cni-plugins-mdg4m" Jul 20 21:02:43.201475 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.199316 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e253d85e-aa94-494c-aaaf-daa7d9490119-etc-sysctl-d\") pod \"tuned-7z98q\" (UID: \"e253d85e-aa94-494c-aaaf-daa7d9490119\") " pod="openshift-cluster-node-tuning-operator/tuned-7z98q" Jul 20 21:02:43.201475 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.199344 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c4188589-18dc-4bd1-9267-688596e53a7d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mdg4m\" (UID: \"c4188589-18dc-4bd1-9267-688596e53a7d\") " pod="openshift-multus/multus-additional-cni-plugins-mdg4m" Jul 20 21:02:43.201475 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.199440 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e253d85e-aa94-494c-aaaf-daa7d9490119-etc-tuned\") pod \"tuned-7z98q\" (UID: \"e253d85e-aa94-494c-aaaf-daa7d9490119\") " pod="openshift-cluster-node-tuning-operator/tuned-7z98q" Jul 20 21:02:43.201475 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.199475 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzztq\" (UniqueName: \"kubernetes.io/projected/e253d85e-aa94-494c-aaaf-daa7d9490119-kube-api-access-fzztq\") pod \"tuned-7z98q\" (UID: \"e253d85e-aa94-494c-aaaf-daa7d9490119\") " pod="openshift-cluster-node-tuning-operator/tuned-7z98q" Jul 20 21:02:43.201475 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.199516 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a853a35b-650d-4ea4-81d3-d4c2ce54e9f1-cnibin\") pod \"multus-vz8z6\" (UID: \"a853a35b-650d-4ea4-81d3-d4c2ce54e9f1\") " pod="openshift-multus/multus-vz8z6" Jul 20 21:02:43.201475 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.199565 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c4188589-18dc-4bd1-9267-688596e53a7d-cni-binary-copy\") pod \"multus-additional-cni-plugins-mdg4m\" (UID: \"c4188589-18dc-4bd1-9267-688596e53a7d\") " pod="openshift-multus/multus-additional-cni-plugins-mdg4m" Jul 20 21:02:43.201475 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.199574 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a853a35b-650d-4ea4-81d3-d4c2ce54e9f1-cnibin\") pod \"multus-vz8z6\" (UID: \"a853a35b-650d-4ea4-81d3-d4c2ce54e9f1\") " pod="openshift-multus/multus-vz8z6" Jul 20 21:02:43.201475 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.199578 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3d2247cc-c3ac-4a33-a206-a40a152d29fe-host-slash\") pod \"iptables-alerter-55bk8\" (UID: \"3d2247cc-c3ac-4a33-a206-a40a152d29fe\") " pod="openshift-network-operator/iptables-alerter-55bk8" Jul 20 21:02:43.201475 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.199633 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3d2247cc-c3ac-4a33-a206-a40a152d29fe-host-slash\") pod \"iptables-alerter-55bk8\" (UID: \"3d2247cc-c3ac-4a33-a206-a40a152d29fe\") " pod="openshift-network-operator/iptables-alerter-55bk8" Jul 20 21:02:43.202068 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.199673 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/68cad356-4b30-41ba-b68f-281c78eb24c1-sys-fs\") pod \"aws-ebs-csi-driver-node-vdpbc\" (UID: \"68cad356-4b30-41ba-b68f-281c78eb24c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdpbc" Jul 20 21:02:43.202068 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.199700 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/356480cb-3559-4be3-a882-45d97dd6e493-host-slash\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.202068 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.199725 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/356480cb-3559-4be3-a882-45d97dd6e493-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.202068 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.199760 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e253d85e-aa94-494c-aaaf-daa7d9490119-host\") pod \"tuned-7z98q\" (UID: \"e253d85e-aa94-494c-aaaf-daa7d9490119\") " pod="openshift-cluster-node-tuning-operator/tuned-7z98q" Jul 20 21:02:43.209429 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.209380 2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 20 21:02:43.213789 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.213700 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb7d5\" (UniqueName: \"kubernetes.io/projected/c4188589-18dc-4bd1-9267-688596e53a7d-kube-api-access-rb7d5\") pod \"multus-additional-cni-plugins-mdg4m\" (UID: \"c4188589-18dc-4bd1-9267-688596e53a7d\") " pod="openshift-multus/multus-additional-cni-plugins-mdg4m" Jul 20 21:02:43.213789 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.213705 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj457\" (UniqueName: \"kubernetes.io/projected/3d2247cc-c3ac-4a33-a206-a40a152d29fe-kube-api-access-vj457\") pod \"iptables-alerter-55bk8\" (UID: \"3d2247cc-c3ac-4a33-a206-a40a152d29fe\") " pod="openshift-network-operator/iptables-alerter-55bk8" Jul 20 21:02:43.213789 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.213703 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq5fr\" (UniqueName: \"kubernetes.io/projected/a853a35b-650d-4ea4-81d3-d4c2ce54e9f1-kube-api-access-rq5fr\") pod \"multus-vz8z6\" (UID: \"a853a35b-650d-4ea4-81d3-d4c2ce54e9f1\") " pod="openshift-multus/multus-vz8z6" Jul 20 21:02:43.213789 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.213700 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mpps\" (UniqueName: \"kubernetes.io/projected/0f09f6b9-6ed0-4165-bc40-706eb42f8f0d-kube-api-access-6mpps\") pod \"network-metrics-daemon-t79t8\" (UID: \"0f09f6b9-6ed0-4165-bc40-706eb42f8f0d\") " pod="openshift-multus/network-metrics-daemon-t79t8" Jul 20 21:02:43.268126 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.268071 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-32.ec2.internal" event={"ID":"6ba459730c951195c7a8ffe9d5d50c7d","Type":"ContainerStarted","Data":"e5b92a7d379088540c616d1707ccb0e555739c65dd2fcf9638548c685349f0ac"} Jul 20 21:02:43.269395 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.269367 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-32.ec2.internal" event={"ID":"197924819a9b1fbec1e5ecce03cf3186","Type":"ContainerStarted","Data":"7759f7ccda9330d0a47d8e2ab522e1c745782dfe98b38e96a6040bfb55238df5"} Jul 20 21:02:43.300502 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.300462 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/356480cb-3559-4be3-a882-45d97dd6e493-node-log\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.300673 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.300511 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/356480cb-3559-4be3-a882-45d97dd6e493-host-kubelet\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.300673 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.300524 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/356480cb-3559-4be3-a882-45d97dd6e493-node-log\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.300673 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.300541 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e253d85e-aa94-494c-aaaf-daa7d9490119-etc-modprobe-d\") pod \"tuned-7z98q\" (UID: \"e253d85e-aa94-494c-aaaf-daa7d9490119\") " pod="openshift-cluster-node-tuning-operator/tuned-7z98q" Jul 20 21:02:43.300673 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.300607 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/68cad356-4b30-41ba-b68f-281c78eb24c1-socket-dir\") pod \"aws-ebs-csi-driver-node-vdpbc\" (UID: \"68cad356-4b30-41ba-b68f-281c78eb24c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdpbc" Jul 20 21:02:43.300673 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.300663 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e253d85e-aa94-494c-aaaf-daa7d9490119-etc-modprobe-d\") pod \"tuned-7z98q\" (UID: \"e253d85e-aa94-494c-aaaf-daa7d9490119\") " pod="openshift-cluster-node-tuning-operator/tuned-7z98q" Jul 20 21:02:43.300951 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.300608 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/356480cb-3559-4be3-a882-45d97dd6e493-host-kubelet\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.300951 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.300649 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dde8ddcb-7c86-4c9d-bd00-11129c8ca35e-tmp-dir\") pod \"node-resolver-qcsfr\" (UID: \"dde8ddcb-7c86-4c9d-bd00-11129c8ca35e\") " pod="openshift-dns/node-resolver-qcsfr" Jul 20 21:02:43.300951 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.300722 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/356480cb-3559-4be3-a882-45d97dd6e493-systemd-units\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.300951 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.300737 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/68cad356-4b30-41ba-b68f-281c78eb24c1-socket-dir\") pod \"aws-ebs-csi-driver-node-vdpbc\" (UID: \"68cad356-4b30-41ba-b68f-281c78eb24c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdpbc" Jul 20 21:02:43.300951 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.300778 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/68cad356-4b30-41ba-b68f-281c78eb24c1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vdpbc\" (UID: \"68cad356-4b30-41ba-b68f-281c78eb24c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdpbc" Jul 20 21:02:43.300951 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.300788 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/356480cb-3559-4be3-a882-45d97dd6e493-systemd-units\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.300951 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.300803 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ef68231f-0506-4450-9fcb-bc14af3b7d76-host\") pod \"node-ca-lgq8n\" (UID: \"ef68231f-0506-4450-9fcb-bc14af3b7d76\") " pod="openshift-image-registry/node-ca-lgq8n" Jul 20 21:02:43.300951 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.300827 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctmx4\" (UniqueName: \"kubernetes.io/projected/356480cb-3559-4be3-a882-45d97dd6e493-kube-api-access-ctmx4\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.300951 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.300844 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ef68231f-0506-4450-9fcb-bc14af3b7d76-host\") pod \"node-ca-lgq8n\" (UID: \"ef68231f-0506-4450-9fcb-bc14af3b7d76\") " pod="openshift-image-registry/node-ca-lgq8n" Jul 20 21:02:43.300951 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.300853 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e253d85e-aa94-494c-aaaf-daa7d9490119-etc-sysctl-d\") pod \"tuned-7z98q\" (UID: \"e253d85e-aa94-494c-aaaf-daa7d9490119\") " pod="openshift-cluster-node-tuning-operator/tuned-7z98q" Jul 20 21:02:43.300951 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.300853 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/68cad356-4b30-41ba-b68f-281c78eb24c1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vdpbc\" (UID: \"68cad356-4b30-41ba-b68f-281c78eb24c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdpbc" Jul 20 21:02:43.300951 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.300877 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e253d85e-aa94-494c-aaaf-daa7d9490119-etc-tuned\") pod \"tuned-7z98q\" (UID: \"e253d85e-aa94-494c-aaaf-daa7d9490119\") " pod="openshift-cluster-node-tuning-operator/tuned-7z98q" Jul 20 21:02:43.300951 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.300913 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzztq\" (UniqueName: \"kubernetes.io/projected/e253d85e-aa94-494c-aaaf-daa7d9490119-kube-api-access-fzztq\") pod \"tuned-7z98q\" (UID: \"e253d85e-aa94-494c-aaaf-daa7d9490119\") " pod="openshift-cluster-node-tuning-operator/tuned-7z98q" Jul 20 21:02:43.300951 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.300920 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dde8ddcb-7c86-4c9d-bd00-11129c8ca35e-tmp-dir\") pod \"node-resolver-qcsfr\" (UID: \"dde8ddcb-7c86-4c9d-bd00-11129c8ca35e\") " pod="openshift-dns/node-resolver-qcsfr" Jul 20 21:02:43.300951 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.300938 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/68cad356-4b30-41ba-b68f-281c78eb24c1-sys-fs\") pod \"aws-ebs-csi-driver-node-vdpbc\" (UID: \"68cad356-4b30-41ba-b68f-281c78eb24c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdpbc" Jul 20 21:02:43.301657 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.300961 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/356480cb-3559-4be3-a882-45d97dd6e493-host-slash\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.301657 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.300977 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e253d85e-aa94-494c-aaaf-daa7d9490119-etc-sysctl-d\") pod \"tuned-7z98q\" (UID: \"e253d85e-aa94-494c-aaaf-daa7d9490119\") " pod="openshift-cluster-node-tuning-operator/tuned-7z98q" Jul 20 21:02:43.301657 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.300986 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/356480cb-3559-4be3-a882-45d97dd6e493-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.301657 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.301010 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e253d85e-aa94-494c-aaaf-daa7d9490119-host\") pod \"tuned-7z98q\" (UID: \"e253d85e-aa94-494c-aaaf-daa7d9490119\") " pod="openshift-cluster-node-tuning-operator/tuned-7z98q" Jul 20 21:02:43.301657 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.301029 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/356480cb-3559-4be3-a882-45d97dd6e493-host-slash\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.301657 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.301034 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/356480cb-3559-4be3-a882-45d97dd6e493-host-run-netns\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.301657 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.301078 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/356480cb-3559-4be3-a882-45d97dd6e493-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.301657 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.301038 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/68cad356-4b30-41ba-b68f-281c78eb24c1-sys-fs\") pod \"aws-ebs-csi-driver-node-vdpbc\" (UID: \"68cad356-4b30-41ba-b68f-281c78eb24c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdpbc" Jul 20 21:02:43.301657 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.301094 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e253d85e-aa94-494c-aaaf-daa7d9490119-host\") pod \"tuned-7z98q\" (UID: \"e253d85e-aa94-494c-aaaf-daa7d9490119\") " pod="openshift-cluster-node-tuning-operator/tuned-7z98q" Jul 20 21:02:43.301657 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.301103 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/356480cb-3559-4be3-a882-45d97dd6e493-host-run-netns\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.301657 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.301092 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gb6r\" (UniqueName: \"kubernetes.io/projected/68cad356-4b30-41ba-b68f-281c78eb24c1-kube-api-access-5gb6r\") pod \"aws-ebs-csi-driver-node-vdpbc\" (UID: \"68cad356-4b30-41ba-b68f-281c78eb24c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdpbc" Jul 20 21:02:43.301657 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.301178 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ef68231f-0506-4450-9fcb-bc14af3b7d76-serviceca\") pod \"node-ca-lgq8n\" (UID: \"ef68231f-0506-4450-9fcb-bc14af3b7d76\") " pod="openshift-image-registry/node-ca-lgq8n" Jul 20 21:02:43.301657 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.301221 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/356480cb-3559-4be3-a882-45d97dd6e493-run-ovn\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.301657 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.301255 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/356480cb-3559-4be3-a882-45d97dd6e493-ovnkube-script-lib\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.302381 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.301280 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e253d85e-aa94-494c-aaaf-daa7d9490119-etc-sysconfig\") pod \"tuned-7z98q\" (UID: \"e253d85e-aa94-494c-aaaf-daa7d9490119\") " pod="openshift-cluster-node-tuning-operator/tuned-7z98q" Jul 20 21:02:43.302381 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.301307 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/356480cb-3559-4be3-a882-45d97dd6e493-var-lib-openvswitch\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.302381 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.301331 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/356480cb-3559-4be3-a882-45d97dd6e493-etc-openvswitch\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.302381 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.301353 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/356480cb-3559-4be3-a882-45d97dd6e493-var-lib-openvswitch\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.302381 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.301361 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e253d85e-aa94-494c-aaaf-daa7d9490119-etc-sysconfig\") pod \"tuned-7z98q\" (UID: \"e253d85e-aa94-494c-aaaf-daa7d9490119\") " pod="openshift-cluster-node-tuning-operator/tuned-7z98q" Jul 20 21:02:43.302381 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.301306 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/356480cb-3559-4be3-a882-45d97dd6e493-run-ovn\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.302381 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.301356 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/356480cb-3559-4be3-a882-45d97dd6e493-host-cni-netd\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.302381 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.301404 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/356480cb-3559-4be3-a882-45d97dd6e493-etc-openvswitch\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.302381 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.301399 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/356480cb-3559-4be3-a882-45d97dd6e493-host-cni-netd\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.302381 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.301426 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/68cad356-4b30-41ba-b68f-281c78eb24c1-registration-dir\") pod \"aws-ebs-csi-driver-node-vdpbc\" (UID: \"68cad356-4b30-41ba-b68f-281c78eb24c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdpbc" Jul 20 21:02:43.302381 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.301452 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/971c4fcc-cdb1-460e-9d0b-6eb07367f5c6-konnectivity-ca\") pod \"konnectivity-agent-5jlkf\" (UID: \"971c4fcc-cdb1-460e-9d0b-6eb07367f5c6\") " pod="kube-system/konnectivity-agent-5jlkf" Jul 20 21:02:43.302381 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.301476 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e253d85e-aa94-494c-aaaf-daa7d9490119-etc-sysctl-conf\") pod \"tuned-7z98q\" (UID: \"e253d85e-aa94-494c-aaaf-daa7d9490119\") " pod="openshift-cluster-node-tuning-operator/tuned-7z98q" Jul 20 21:02:43.302381 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.301499 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e253d85e-aa94-494c-aaaf-daa7d9490119-var-lib-kubelet\") pod \"tuned-7z98q\" (UID: \"e253d85e-aa94-494c-aaaf-daa7d9490119\") " pod="openshift-cluster-node-tuning-operator/tuned-7z98q" Jul 20 21:02:43.302381 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.301522 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzz2n\" (UniqueName: \"kubernetes.io/projected/f0fef424-8965-40f7-8b13-5a72773f050a-kube-api-access-hzz2n\") pod \"network-check-target-5chx7\" (UID: \"f0fef424-8965-40f7-8b13-5a72773f050a\") " pod="openshift-network-diagnostics/network-check-target-5chx7" Jul 20 21:02:43.302978 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.301536 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/68cad356-4b30-41ba-b68f-281c78eb24c1-registration-dir\") pod \"aws-ebs-csi-driver-node-vdpbc\" (UID: \"68cad356-4b30-41ba-b68f-281c78eb24c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdpbc" Jul 20 21:02:43.302978 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.301547 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/356480cb-3559-4be3-a882-45d97dd6e493-host-cni-bin\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.302978 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.301594 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/356480cb-3559-4be3-a882-45d97dd6e493-host-cni-bin\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.302978 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.301594 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e253d85e-aa94-494c-aaaf-daa7d9490119-var-lib-kubelet\") pod \"tuned-7z98q\" (UID: \"e253d85e-aa94-494c-aaaf-daa7d9490119\") " pod="openshift-cluster-node-tuning-operator/tuned-7z98q" Jul 20 21:02:43.302978 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.301613 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/356480cb-3559-4be3-a882-45d97dd6e493-env-overrides\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.302978 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.301644 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/356480cb-3559-4be3-a882-45d97dd6e493-run-systemd\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.302978 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.301666 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/356480cb-3559-4be3-a882-45d97dd6e493-log-socket\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.302978 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.301687 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ef68231f-0506-4450-9fcb-bc14af3b7d76-serviceca\") pod \"node-ca-lgq8n\" (UID: \"ef68231f-0506-4450-9fcb-bc14af3b7d76\") " pod="openshift-image-registry/node-ca-lgq8n" Jul 20 21:02:43.302978 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.301692 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/356480cb-3559-4be3-a882-45d97dd6e493-ovn-node-metrics-cert\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.302978 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.301715 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e253d85e-aa94-494c-aaaf-daa7d9490119-lib-modules\") pod \"tuned-7z98q\" (UID: \"e253d85e-aa94-494c-aaaf-daa7d9490119\") " pod="openshift-cluster-node-tuning-operator/tuned-7z98q" Jul 20 21:02:43.302978 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.301738 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/971c4fcc-cdb1-460e-9d0b-6eb07367f5c6-agent-certs\") pod \"konnectivity-agent-5jlkf\" (UID: \"971c4fcc-cdb1-460e-9d0b-6eb07367f5c6\") " pod="kube-system/konnectivity-agent-5jlkf" Jul 20 21:02:43.302978 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.301769 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/68cad356-4b30-41ba-b68f-281c78eb24c1-etc-selinux\") pod \"aws-ebs-csi-driver-node-vdpbc\" (UID: \"68cad356-4b30-41ba-b68f-281c78eb24c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdpbc" Jul 20 21:02:43.302978 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.301787 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/356480cb-3559-4be3-a882-45d97dd6e493-ovnkube-script-lib\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.302978 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.301795 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/356480cb-3559-4be3-a882-45d97dd6e493-ovnkube-config\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.302978 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.301818 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e253d85e-aa94-494c-aaaf-daa7d9490119-etc-systemd\") pod \"tuned-7z98q\" (UID: \"e253d85e-aa94-494c-aaaf-daa7d9490119\") " pod="openshift-cluster-node-tuning-operator/tuned-7z98q" Jul 20 21:02:43.303692 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.301823 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e253d85e-aa94-494c-aaaf-daa7d9490119-etc-sysctl-conf\") pod \"tuned-7z98q\" (UID: \"e253d85e-aa94-494c-aaaf-daa7d9490119\") " pod="openshift-cluster-node-tuning-operator/tuned-7z98q" Jul 20 21:02:43.303692 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.301843 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f8gv\" (UniqueName: \"kubernetes.io/projected/ef68231f-0506-4450-9fcb-bc14af3b7d76-kube-api-access-6f8gv\") pod \"node-ca-lgq8n\" (UID: \"ef68231f-0506-4450-9fcb-bc14af3b7d76\") " pod="openshift-image-registry/node-ca-lgq8n" Jul 20 21:02:43.303692 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.301868 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/356480cb-3559-4be3-a882-45d97dd6e493-run-openvswitch\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.303692 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.301893 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e253d85e-aa94-494c-aaaf-daa7d9490119-run\") pod \"tuned-7z98q\" (UID: \"e253d85e-aa94-494c-aaaf-daa7d9490119\") " pod="openshift-cluster-node-tuning-operator/tuned-7z98q" Jul 20 21:02:43.303692 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.301897 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e253d85e-aa94-494c-aaaf-daa7d9490119-lib-modules\") pod \"tuned-7z98q\" (UID: \"e253d85e-aa94-494c-aaaf-daa7d9490119\") " pod="openshift-cluster-node-tuning-operator/tuned-7z98q" Jul 20 21:02:43.303692 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.301899 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/68cad356-4b30-41ba-b68f-281c78eb24c1-etc-selinux\") pod \"aws-ebs-csi-driver-node-vdpbc\" (UID: \"68cad356-4b30-41ba-b68f-281c78eb24c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdpbc" Jul 20 21:02:43.303692 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.301964 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e253d85e-aa94-494c-aaaf-daa7d9490119-sys\") pod \"tuned-7z98q\" (UID: \"e253d85e-aa94-494c-aaaf-daa7d9490119\") " pod="openshift-cluster-node-tuning-operator/tuned-7z98q" Jul 20 21:02:43.303692 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.302006 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/356480cb-3559-4be3-a882-45d97dd6e493-log-socket\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.303692 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.302012 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/971c4fcc-cdb1-460e-9d0b-6eb07367f5c6-konnectivity-ca\") pod \"konnectivity-agent-5jlkf\" (UID: \"971c4fcc-cdb1-460e-9d0b-6eb07367f5c6\") " pod="kube-system/konnectivity-agent-5jlkf" Jul 20 21:02:43.303692 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.301738 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/356480cb-3559-4be3-a882-45d97dd6e493-run-systemd\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.303692 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.301914 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e253d85e-aa94-494c-aaaf-daa7d9490119-sys\") pod \"tuned-7z98q\" (UID: \"e253d85e-aa94-494c-aaaf-daa7d9490119\") " pod="openshift-cluster-node-tuning-operator/tuned-7z98q" Jul 20 21:02:43.303692 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.302074 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/356480cb-3559-4be3-a882-45d97dd6e493-env-overrides\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.303692 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.302098 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e253d85e-aa94-494c-aaaf-daa7d9490119-tmp\") pod \"tuned-7z98q\" (UID: \"e253d85e-aa94-494c-aaaf-daa7d9490119\") " pod="openshift-cluster-node-tuning-operator/tuned-7z98q" Jul 20 21:02:43.303692 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.302122 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dde8ddcb-7c86-4c9d-bd00-11129c8ca35e-hosts-file\") pod \"node-resolver-qcsfr\" (UID: \"dde8ddcb-7c86-4c9d-bd00-11129c8ca35e\") " pod="openshift-dns/node-resolver-qcsfr" Jul 20 21:02:43.303692 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.302177 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq7k9\" (UniqueName: \"kubernetes.io/projected/dde8ddcb-7c86-4c9d-bd00-11129c8ca35e-kube-api-access-pq7k9\") pod \"node-resolver-qcsfr\" (UID: \"dde8ddcb-7c86-4c9d-bd00-11129c8ca35e\") " pod="openshift-dns/node-resolver-qcsfr" Jul 20 21:02:43.304445 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.302204 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/68cad356-4b30-41ba-b68f-281c78eb24c1-device-dir\") pod \"aws-ebs-csi-driver-node-vdpbc\" (UID: \"68cad356-4b30-41ba-b68f-281c78eb24c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdpbc" Jul 20 21:02:43.304445 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.302231 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/356480cb-3559-4be3-a882-45d97dd6e493-host-run-ovn-kubernetes\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.304445 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.302257 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e253d85e-aa94-494c-aaaf-daa7d9490119-etc-kubernetes\") pod \"tuned-7z98q\" (UID: \"e253d85e-aa94-494c-aaaf-daa7d9490119\") " pod="openshift-cluster-node-tuning-operator/tuned-7z98q" Jul 20 21:02:43.304445 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.302275 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e253d85e-aa94-494c-aaaf-daa7d9490119-etc-systemd\") pod \"tuned-7z98q\" (UID: \"e253d85e-aa94-494c-aaaf-daa7d9490119\") " pod="openshift-cluster-node-tuning-operator/tuned-7z98q" Jul 20 21:02:43.304445 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.302326 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dde8ddcb-7c86-4c9d-bd00-11129c8ca35e-hosts-file\") pod \"node-resolver-qcsfr\" (UID: \"dde8ddcb-7c86-4c9d-bd00-11129c8ca35e\") " pod="openshift-dns/node-resolver-qcsfr" Jul 20 21:02:43.304445 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.302359 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/356480cb-3559-4be3-a882-45d97dd6e493-run-openvswitch\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.304445 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.302363 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e253d85e-aa94-494c-aaaf-daa7d9490119-etc-kubernetes\") pod \"tuned-7z98q\" (UID: \"e253d85e-aa94-494c-aaaf-daa7d9490119\") " pod="openshift-cluster-node-tuning-operator/tuned-7z98q" Jul 20 21:02:43.304445 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.302400 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/356480cb-3559-4be3-a882-45d97dd6e493-host-run-ovn-kubernetes\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.304445 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.302453 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/68cad356-4b30-41ba-b68f-281c78eb24c1-device-dir\") pod \"aws-ebs-csi-driver-node-vdpbc\" (UID: \"68cad356-4b30-41ba-b68f-281c78eb24c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdpbc" Jul 20 21:02:43.304445 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.302458 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e253d85e-aa94-494c-aaaf-daa7d9490119-run\") pod \"tuned-7z98q\" (UID: \"e253d85e-aa94-494c-aaaf-daa7d9490119\") " pod="openshift-cluster-node-tuning-operator/tuned-7z98q" Jul 20 21:02:43.304445 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.302633 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/356480cb-3559-4be3-a882-45d97dd6e493-ovnkube-config\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.304445 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.304414 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e253d85e-aa94-494c-aaaf-daa7d9490119-etc-tuned\") pod \"tuned-7z98q\" (UID: \"e253d85e-aa94-494c-aaaf-daa7d9490119\") " pod="openshift-cluster-node-tuning-operator/tuned-7z98q" Jul 20 21:02:43.305010 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.304643 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/356480cb-3559-4be3-a882-45d97dd6e493-ovn-node-metrics-cert\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.305247 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.305226 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/971c4fcc-cdb1-460e-9d0b-6eb07367f5c6-agent-certs\") pod \"konnectivity-agent-5jlkf\" (UID: \"971c4fcc-cdb1-460e-9d0b-6eb07367f5c6\") " pod="kube-system/konnectivity-agent-5jlkf" Jul 20 21:02:43.306356 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.306332 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e253d85e-aa94-494c-aaaf-daa7d9490119-tmp\") pod \"tuned-7z98q\" (UID: \"e253d85e-aa94-494c-aaaf-daa7d9490119\") " pod="openshift-cluster-node-tuning-operator/tuned-7z98q" Jul 20 21:02:43.310893 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:43.310871 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 21:02:43.310893 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:43.310896 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 21:02:43.311041 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:43.310908 2562 projected.go:196] Error preparing data for projected volume kube-api-access-hzz2n for pod openshift-network-diagnostics/network-check-target-5chx7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 21:02:43.311041 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:43.310973 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f0fef424-8965-40f7-8b13-5a72773f050a-kube-api-access-hzz2n podName:f0fef424-8965-40f7-8b13-5a72773f050a nodeName:}" failed. No retries permitted until 2026-07-20 21:02:43.810956574 +0000 UTC m=+3.136659495 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hzz2n" (UniqueName: "kubernetes.io/projected/f0fef424-8965-40f7-8b13-5a72773f050a-kube-api-access-hzz2n") pod "network-check-target-5chx7" (UID: "f0fef424-8965-40f7-8b13-5a72773f050a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 21:02:43.313111 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.313059 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gb6r\" (UniqueName: \"kubernetes.io/projected/68cad356-4b30-41ba-b68f-281c78eb24c1-kube-api-access-5gb6r\") pod \"aws-ebs-csi-driver-node-vdpbc\" (UID: \"68cad356-4b30-41ba-b68f-281c78eb24c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdpbc" Jul 20 21:02:43.314635 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.314283 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzztq\" (UniqueName: \"kubernetes.io/projected/e253d85e-aa94-494c-aaaf-daa7d9490119-kube-api-access-fzztq\") pod \"tuned-7z98q\" (UID: \"e253d85e-aa94-494c-aaaf-daa7d9490119\") " pod="openshift-cluster-node-tuning-operator/tuned-7z98q" Jul 20 21:02:43.314635 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.314316 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq7k9\" (UniqueName: \"kubernetes.io/projected/dde8ddcb-7c86-4c9d-bd00-11129c8ca35e-kube-api-access-pq7k9\") pod \"node-resolver-qcsfr\" (UID: \"dde8ddcb-7c86-4c9d-bd00-11129c8ca35e\") " pod="openshift-dns/node-resolver-qcsfr" Jul 20 21:02:43.317580 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.317559 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctmx4\" (UniqueName: \"kubernetes.io/projected/356480cb-3559-4be3-a882-45d97dd6e493-kube-api-access-ctmx4\") pod \"ovnkube-node-9x7cd\" (UID: \"356480cb-3559-4be3-a882-45d97dd6e493\") " pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.317685 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.317663 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f8gv\" (UniqueName: \"kubernetes.io/projected/ef68231f-0506-4450-9fcb-bc14af3b7d76-kube-api-access-6f8gv\") pod \"node-ca-lgq8n\" (UID: \"ef68231f-0506-4450-9fcb-bc14af3b7d76\") " pod="openshift-image-registry/node-ca-lgq8n" Jul 20 21:02:43.332310 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.332278 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 21:02:43.394543 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.394506 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mdg4m" Jul 20 21:02:43.401185 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.401166 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vz8z6" Jul 20 21:02:43.401302 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.401168 2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 21:02:43.409247 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.409217 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-55bk8" Jul 20 21:02:43.413984 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.413963 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:02:43.426211 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.426190 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7z98q" Jul 20 21:02:43.434168 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.434128 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdpbc" Jul 20 21:02:43.441893 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.441874 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5jlkf" Jul 20 21:02:43.449500 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.449471 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lgq8n" Jul 20 21:02:43.456628 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.456605 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qcsfr" Jul 20 21:02:43.704633 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.704541 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b3600ee-a2f5-4b18-ac22-9c868136ce69-original-pull-secret\") pod \"global-pull-secret-syncer-2lqmn\" (UID: \"2b3600ee-a2f5-4b18-ac22-9c868136ce69\") " pod="kube-system/global-pull-secret-syncer-2lqmn" Jul 20 21:02:43.704633 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.704601 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f09f6b9-6ed0-4165-bc40-706eb42f8f0d-metrics-certs\") pod \"network-metrics-daemon-t79t8\" (UID: \"0f09f6b9-6ed0-4165-bc40-706eb42f8f0d\") " pod="openshift-multus/network-metrics-daemon-t79t8" Jul 20 21:02:43.704846 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:43.704719 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 21:02:43.704846 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:43.704752 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 21:02:43.704846 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:43.704816 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b3600ee-a2f5-4b18-ac22-9c868136ce69-original-pull-secret podName:2b3600ee-a2f5-4b18-ac22-9c868136ce69 nodeName:}" failed. No retries permitted until 2026-07-20 21:02:44.704795667 +0000 UTC m=+4.030498601 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b3600ee-a2f5-4b18-ac22-9c868136ce69-original-pull-secret") pod "global-pull-secret-syncer-2lqmn" (UID: "2b3600ee-a2f5-4b18-ac22-9c868136ce69") : object "kube-system"/"original-pull-secret" not registered Jul 20 21:02:43.704994 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:43.704837 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f09f6b9-6ed0-4165-bc40-706eb42f8f0d-metrics-certs podName:0f09f6b9-6ed0-4165-bc40-706eb42f8f0d nodeName:}" failed. No retries permitted until 2026-07-20 21:02:44.704826031 +0000 UTC m=+4.030528958 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0f09f6b9-6ed0-4165-bc40-706eb42f8f0d-metrics-certs") pod "network-metrics-daemon-t79t8" (UID: "0f09f6b9-6ed0-4165-bc40-706eb42f8f0d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 21:02:43.905536 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:43.905497 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzz2n\" (UniqueName: \"kubernetes.io/projected/f0fef424-8965-40f7-8b13-5a72773f050a-kube-api-access-hzz2n\") pod \"network-check-target-5chx7\" (UID: \"f0fef424-8965-40f7-8b13-5a72773f050a\") " pod="openshift-network-diagnostics/network-check-target-5chx7" Jul 20 21:02:43.905709 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:43.905690 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 21:02:43.905760 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:43.905720 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 21:02:43.905760 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:43.905735 2562 projected.go:196] Error preparing data for projected volume kube-api-access-hzz2n for pod openshift-network-diagnostics/network-check-target-5chx7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 21:02:43.905821 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:43.905795 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f0fef424-8965-40f7-8b13-5a72773f050a-kube-api-access-hzz2n podName:f0fef424-8965-40f7-8b13-5a72773f050a nodeName:}" failed. No retries permitted until 2026-07-20 21:02:44.905778061 +0000 UTC m=+4.231480987 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hzz2n" (UniqueName: "kubernetes.io/projected/f0fef424-8965-40f7-8b13-5a72773f050a-kube-api-access-hzz2n") pod "network-check-target-5chx7" (UID: "f0fef424-8965-40f7-8b13-5a72773f050a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 21:02:44.126068 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:44.125999 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d2247cc_c3ac_4a33_a206_a40a152d29fe.slice/crio-17f59a92f825c4150f8667b50eed7bd462405be776871c32ae1585c4fefe471b WatchSource:0}: Error finding container 17f59a92f825c4150f8667b50eed7bd462405be776871c32ae1585c4fefe471b: Status 404 returned error can't find the container with id 17f59a92f825c4150f8667b50eed7bd462405be776871c32ae1585c4fefe471b Jul 20 21:02:44.127394 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:44.127362 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4188589_18dc_4bd1_9267_688596e53a7d.slice/crio-205d3856d1bc772e333c883341ea47836356eb818047e29cb75a7c274f868a2f WatchSource:0}: Error finding container 205d3856d1bc772e333c883341ea47836356eb818047e29cb75a7c274f868a2f: Status 404 returned error can't find the container with id 205d3856d1bc772e333c883341ea47836356eb818047e29cb75a7c274f868a2f Jul 20 21:02:44.128286 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:44.128258 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda853a35b_650d_4ea4_81d3_d4c2ce54e9f1.slice/crio-a5fec3d70d07f7efd9e92949edfdfbf6beba4778e4c8c9aaaf07b889c7e67e1a WatchSource:0}: Error finding container a5fec3d70d07f7efd9e92949edfdfbf6beba4778e4c8c9aaaf07b889c7e67e1a: Status 404 returned error can't find the container with id a5fec3d70d07f7efd9e92949edfdfbf6beba4778e4c8c9aaaf07b889c7e67e1a Jul 20 21:02:44.130427 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:44.130401 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod971c4fcc_cdb1_460e_9d0b_6eb07367f5c6.slice/crio-fca21d32ea7397f36c25bf84bb1637469570792445596288ff0ef88c0eb49864 WatchSource:0}: Error finding container fca21d32ea7397f36c25bf84bb1637469570792445596288ff0ef88c0eb49864: Status 404 returned error can't find the container with id fca21d32ea7397f36c25bf84bb1637469570792445596288ff0ef88c0eb49864 Jul 20 21:02:44.132739 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:44.132715 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddde8ddcb_7c86_4c9d_bd00_11129c8ca35e.slice/crio-edd0b29930f453bc72a06ecdcd18733c3931ee34ae0fc863a916bfb3b0375553 WatchSource:0}: Error finding container edd0b29930f453bc72a06ecdcd18733c3931ee34ae0fc863a916bfb3b0375553: Status 404 returned error can't find the container with id edd0b29930f453bc72a06ecdcd18733c3931ee34ae0fc863a916bfb3b0375553 Jul 20 21:02:44.133656 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:44.133618 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef68231f_0506_4450_9fcb_bc14af3b7d76.slice/crio-f284ffaf8de87d9cbe2bedd7e3a9de8116cb3f1f6b7a4194fc2d61e2e2bc619c WatchSource:0}: Error finding container f284ffaf8de87d9cbe2bedd7e3a9de8116cb3f1f6b7a4194fc2d61e2e2bc619c: Status 404 returned error can't find the container with id f284ffaf8de87d9cbe2bedd7e3a9de8116cb3f1f6b7a4194fc2d61e2e2bc619c Jul 20 21:02:44.134805 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:44.134583 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod356480cb_3559_4be3_a882_45d97dd6e493.slice/crio-0370e437667e6f9f08a6a0f779810459eec7a001869bee79c7e0ed66edaf1d51 WatchSource:0}: Error finding container 0370e437667e6f9f08a6a0f779810459eec7a001869bee79c7e0ed66edaf1d51: Status 404 returned error can't find the container with id 0370e437667e6f9f08a6a0f779810459eec7a001869bee79c7e0ed66edaf1d51 Jul 20 21:02:44.137561 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:44.135586 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode253d85e_aa94_494c_aaaf_daa7d9490119.slice/crio-bf1cc0fa9a7cfa535ee8c21f7bb3126a64277c6ad7833404709b1f8f881a5884 WatchSource:0}: Error finding container bf1cc0fa9a7cfa535ee8c21f7bb3126a64277c6ad7833404709b1f8f881a5884: Status 404 returned error can't find the container with id bf1cc0fa9a7cfa535ee8c21f7bb3126a64277c6ad7833404709b1f8f881a5884 Jul 20 21:02:44.137561 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:02:44.137049 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68cad356_4b30_41ba_b68f_281c78eb24c1.slice/crio-833a0daa0fffaa4229ac6a0289a2e763338cedf5fd829debc02ad68ffa9ae3d3 WatchSource:0}: Error finding container 833a0daa0fffaa4229ac6a0289a2e763338cedf5fd829debc02ad68ffa9ae3d3: Status 404 returned error can't find the container with id 833a0daa0fffaa4229ac6a0289a2e763338cedf5fd829debc02ad68ffa9ae3d3 Jul 20 21:02:44.152792 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:44.152752 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-19 20:57:43 +0000 UTC" deadline="2027-12-31 10:08:42.773002558 +0000 UTC" Jul 20 21:02:44.152792 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:44.152789 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12685h5m58.620216118s" Jul 20 21:02:44.271795 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:44.271765 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lgq8n" event={"ID":"ef68231f-0506-4450-9fcb-bc14af3b7d76","Type":"ContainerStarted","Data":"f284ffaf8de87d9cbe2bedd7e3a9de8116cb3f1f6b7a4194fc2d61e2e2bc619c"} Jul 20 21:02:44.273685 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:44.273659 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7z98q" event={"ID":"e253d85e-aa94-494c-aaaf-daa7d9490119","Type":"ContainerStarted","Data":"bf1cc0fa9a7cfa535ee8c21f7bb3126a64277c6ad7833404709b1f8f881a5884"} Jul 20 21:02:44.274659 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:44.274633 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdpbc" event={"ID":"68cad356-4b30-41ba-b68f-281c78eb24c1","Type":"ContainerStarted","Data":"833a0daa0fffaa4229ac6a0289a2e763338cedf5fd829debc02ad68ffa9ae3d3"} Jul 20 21:02:44.276893 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:44.276870 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" event={"ID":"356480cb-3559-4be3-a882-45d97dd6e493","Type":"ContainerStarted","Data":"0370e437667e6f9f08a6a0f779810459eec7a001869bee79c7e0ed66edaf1d51"} Jul 20 21:02:44.277905 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:44.277884 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qcsfr" event={"ID":"dde8ddcb-7c86-4c9d-bd00-11129c8ca35e","Type":"ContainerStarted","Data":"edd0b29930f453bc72a06ecdcd18733c3931ee34ae0fc863a916bfb3b0375553"} Jul 20 21:02:44.278894 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:44.278875 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mdg4m" event={"ID":"c4188589-18dc-4bd1-9267-688596e53a7d","Type":"ContainerStarted","Data":"205d3856d1bc772e333c883341ea47836356eb818047e29cb75a7c274f868a2f"} Jul 20 21:02:44.280685 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:44.280663 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-32.ec2.internal" event={"ID":"6ba459730c951195c7a8ffe9d5d50c7d","Type":"ContainerStarted","Data":"138db6f579b7b3066c7deb042e898a2984c7e12fff6d092c35e9eb17552eff70"} Jul 20 21:02:44.281740 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:44.281722 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5jlkf" event={"ID":"971c4fcc-cdb1-460e-9d0b-6eb07367f5c6","Type":"ContainerStarted","Data":"fca21d32ea7397f36c25bf84bb1637469570792445596288ff0ef88c0eb49864"} Jul 20 21:02:44.282720 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:44.282698 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vz8z6" event={"ID":"a853a35b-650d-4ea4-81d3-d4c2ce54e9f1","Type":"ContainerStarted","Data":"a5fec3d70d07f7efd9e92949edfdfbf6beba4778e4c8c9aaaf07b889c7e67e1a"} Jul 20 21:02:44.283794 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:44.283774 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-55bk8" event={"ID":"3d2247cc-c3ac-4a33-a206-a40a152d29fe","Type":"ContainerStarted","Data":"17f59a92f825c4150f8667b50eed7bd462405be776871c32ae1585c4fefe471b"} Jul 20 21:02:44.715169 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:44.714455 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b3600ee-a2f5-4b18-ac22-9c868136ce69-original-pull-secret\") pod \"global-pull-secret-syncer-2lqmn\" (UID: \"2b3600ee-a2f5-4b18-ac22-9c868136ce69\") " pod="kube-system/global-pull-secret-syncer-2lqmn" Jul 20 21:02:44.715169 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:44.714511 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f09f6b9-6ed0-4165-bc40-706eb42f8f0d-metrics-certs\") pod \"network-metrics-daemon-t79t8\" (UID: \"0f09f6b9-6ed0-4165-bc40-706eb42f8f0d\") " pod="openshift-multus/network-metrics-daemon-t79t8" Jul 20 21:02:44.715169 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:44.714630 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 21:02:44.715169 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:44.714691 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f09f6b9-6ed0-4165-bc40-706eb42f8f0d-metrics-certs podName:0f09f6b9-6ed0-4165-bc40-706eb42f8f0d nodeName:}" failed. No retries permitted until 2026-07-20 21:02:46.714674102 +0000 UTC m=+6.040377026 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0f09f6b9-6ed0-4165-bc40-706eb42f8f0d-metrics-certs") pod "network-metrics-daemon-t79t8" (UID: "0f09f6b9-6ed0-4165-bc40-706eb42f8f0d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 21:02:44.715169 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:44.715074 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 21:02:44.715169 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:44.715118 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b3600ee-a2f5-4b18-ac22-9c868136ce69-original-pull-secret podName:2b3600ee-a2f5-4b18-ac22-9c868136ce69 nodeName:}" failed. No retries permitted until 2026-07-20 21:02:46.715104549 +0000 UTC m=+6.040807477 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b3600ee-a2f5-4b18-ac22-9c868136ce69-original-pull-secret") pod "global-pull-secret-syncer-2lqmn" (UID: "2b3600ee-a2f5-4b18-ac22-9c868136ce69") : object "kube-system"/"original-pull-secret" not registered Jul 20 21:02:44.915752 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:44.915713 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzz2n\" (UniqueName: \"kubernetes.io/projected/f0fef424-8965-40f7-8b13-5a72773f050a-kube-api-access-hzz2n\") pod \"network-check-target-5chx7\" (UID: \"f0fef424-8965-40f7-8b13-5a72773f050a\") " pod="openshift-network-diagnostics/network-check-target-5chx7" Jul 20 21:02:44.915926 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:44.915900 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 21:02:44.915926 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:44.915920 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 21:02:44.916045 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:44.915932 2562 projected.go:196] Error preparing data for projected volume kube-api-access-hzz2n for pod openshift-network-diagnostics/network-check-target-5chx7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 21:02:44.916045 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:44.915990 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f0fef424-8965-40f7-8b13-5a72773f050a-kube-api-access-hzz2n podName:f0fef424-8965-40f7-8b13-5a72773f050a nodeName:}" failed. No retries permitted until 2026-07-20 21:02:46.915972035 +0000 UTC m=+6.241674958 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hzz2n" (UniqueName: "kubernetes.io/projected/f0fef424-8965-40f7-8b13-5a72773f050a-kube-api-access-hzz2n") pod "network-check-target-5chx7" (UID: "f0fef424-8965-40f7-8b13-5a72773f050a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 21:02:45.153734 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:45.153684 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-19 20:57:43 +0000 UTC" deadline="2028-03-21 11:37:24.376093986 +0000 UTC" Jul 20 21:02:45.153734 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:45.153733 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14630h34m39.222366213s" Jul 20 21:02:45.273580 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:45.272231 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2lqmn" Jul 20 21:02:45.273580 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:45.272721 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2lqmn" podUID="2b3600ee-a2f5-4b18-ac22-9c868136ce69" Jul 20 21:02:45.284323 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:45.284284 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5chx7" Jul 20 21:02:45.284501 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:45.284449 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5chx7" podUID="f0fef424-8965-40f7-8b13-5a72773f050a" Jul 20 21:02:45.284570 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:45.284556 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t79t8" Jul 20 21:02:45.284716 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:45.284682 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t79t8" podUID="0f09f6b9-6ed0-4165-bc40-706eb42f8f0d" Jul 20 21:02:45.302685 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:45.302646 2562 generic.go:356] "Generic (PLEG): container finished" podID="197924819a9b1fbec1e5ecce03cf3186" containerID="589dddc9e2baaa495071b9683b08f320e31847c330d119536428a079d7f39140" exitCode=0 Jul 20 21:02:45.303772 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:45.303743 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-32.ec2.internal" event={"ID":"197924819a9b1fbec1e5ecce03cf3186","Type":"ContainerDied","Data":"589dddc9e2baaa495071b9683b08f320e31847c330d119536428a079d7f39140"} Jul 20 21:02:45.323295 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:45.323232 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-32.ec2.internal" podStartSLOduration=3.323210723 podStartE2EDuration="3.323210723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 21:02:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:02:44.296252128 +0000 UTC m=+3.621955070" watchObservedRunningTime="2026-07-20 21:02:45.323210723 +0000 UTC m=+4.648913667" Jul 20 21:02:46.330559 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:46.330512 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-32.ec2.internal" event={"ID":"197924819a9b1fbec1e5ecce03cf3186","Type":"ContainerStarted","Data":"1679049bf8a16660859f72f5a1996f33e0464ace20918d561065f93df5dd6e7c"} Jul 20 21:02:46.735000 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:46.734193 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f09f6b9-6ed0-4165-bc40-706eb42f8f0d-metrics-certs\") pod \"network-metrics-daemon-t79t8\" (UID: \"0f09f6b9-6ed0-4165-bc40-706eb42f8f0d\") " pod="openshift-multus/network-metrics-daemon-t79t8" Jul 20 21:02:46.735000 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:46.734289 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b3600ee-a2f5-4b18-ac22-9c868136ce69-original-pull-secret\") pod \"global-pull-secret-syncer-2lqmn\" (UID: \"2b3600ee-a2f5-4b18-ac22-9c868136ce69\") " pod="kube-system/global-pull-secret-syncer-2lqmn" Jul 20 21:02:46.735000 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:46.734419 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 21:02:46.735000 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:46.734481 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b3600ee-a2f5-4b18-ac22-9c868136ce69-original-pull-secret podName:2b3600ee-a2f5-4b18-ac22-9c868136ce69 nodeName:}" failed. No retries permitted until 2026-07-20 21:02:50.734461486 +0000 UTC m=+10.060164413 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b3600ee-a2f5-4b18-ac22-9c868136ce69-original-pull-secret") pod "global-pull-secret-syncer-2lqmn" (UID: "2b3600ee-a2f5-4b18-ac22-9c868136ce69") : object "kube-system"/"original-pull-secret" not registered Jul 20 21:02:46.735000 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:46.734867 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 21:02:46.735000 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:46.734916 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f09f6b9-6ed0-4165-bc40-706eb42f8f0d-metrics-certs podName:0f09f6b9-6ed0-4165-bc40-706eb42f8f0d nodeName:}" failed. No retries permitted until 2026-07-20 21:02:50.73490107 +0000 UTC m=+10.060603997 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0f09f6b9-6ed0-4165-bc40-706eb42f8f0d-metrics-certs") pod "network-metrics-daemon-t79t8" (UID: "0f09f6b9-6ed0-4165-bc40-706eb42f8f0d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 21:02:46.937403 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:46.936524 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzz2n\" (UniqueName: \"kubernetes.io/projected/f0fef424-8965-40f7-8b13-5a72773f050a-kube-api-access-hzz2n\") pod \"network-check-target-5chx7\" (UID: \"f0fef424-8965-40f7-8b13-5a72773f050a\") " pod="openshift-network-diagnostics/network-check-target-5chx7" Jul 20 21:02:46.937403 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:46.936699 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 21:02:46.937403 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:46.936718 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 21:02:46.937403 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:46.936731 2562 projected.go:196] Error preparing data for projected volume kube-api-access-hzz2n for pod openshift-network-diagnostics/network-check-target-5chx7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 21:02:46.937403 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:46.936791 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f0fef424-8965-40f7-8b13-5a72773f050a-kube-api-access-hzz2n podName:f0fef424-8965-40f7-8b13-5a72773f050a nodeName:}" failed. No retries permitted until 2026-07-20 21:02:50.936772201 +0000 UTC m=+10.262475143 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hzz2n" (UniqueName: "kubernetes.io/projected/f0fef424-8965-40f7-8b13-5a72773f050a-kube-api-access-hzz2n") pod "network-check-target-5chx7" (UID: "f0fef424-8965-40f7-8b13-5a72773f050a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 21:02:47.266306 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:47.266277 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5chx7" Jul 20 21:02:47.266487 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:47.266392 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5chx7" podUID="f0fef424-8965-40f7-8b13-5a72773f050a" Jul 20 21:02:47.266767 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:47.266748 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2lqmn" Jul 20 21:02:47.266879 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:47.266840 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2lqmn" podUID="2b3600ee-a2f5-4b18-ac22-9c868136ce69" Jul 20 21:02:47.266947 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:47.266890 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t79t8" Jul 20 21:02:47.267004 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:47.266974 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t79t8" podUID="0f09f6b9-6ed0-4165-bc40-706eb42f8f0d" Jul 20 21:02:49.264518 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:49.264444 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t79t8" Jul 20 21:02:49.264909 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:49.264562 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5chx7" Jul 20 21:02:49.264909 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:49.264572 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t79t8" podUID="0f09f6b9-6ed0-4165-bc40-706eb42f8f0d" Jul 20 21:02:49.264909 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:49.264689 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5chx7" podUID="f0fef424-8965-40f7-8b13-5a72773f050a" Jul 20 21:02:49.264909 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:49.264738 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2lqmn" Jul 20 21:02:49.264909 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:49.264806 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2lqmn" podUID="2b3600ee-a2f5-4b18-ac22-9c868136ce69" Jul 20 21:02:50.768478 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:50.768425 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b3600ee-a2f5-4b18-ac22-9c868136ce69-original-pull-secret\") pod \"global-pull-secret-syncer-2lqmn\" (UID: \"2b3600ee-a2f5-4b18-ac22-9c868136ce69\") " pod="kube-system/global-pull-secret-syncer-2lqmn" Jul 20 21:02:50.768478 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:50.768490 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f09f6b9-6ed0-4165-bc40-706eb42f8f0d-metrics-certs\") pod \"network-metrics-daemon-t79t8\" (UID: \"0f09f6b9-6ed0-4165-bc40-706eb42f8f0d\") " pod="openshift-multus/network-metrics-daemon-t79t8" Jul 20 21:02:50.768944 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:50.768602 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 21:02:50.768944 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:50.768627 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 21:02:50.768944 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:50.768689 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b3600ee-a2f5-4b18-ac22-9c868136ce69-original-pull-secret podName:2b3600ee-a2f5-4b18-ac22-9c868136ce69 nodeName:}" failed. No retries permitted until 2026-07-20 21:02:58.768667081 +0000 UTC m=+18.094370005 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b3600ee-a2f5-4b18-ac22-9c868136ce69-original-pull-secret") pod "global-pull-secret-syncer-2lqmn" (UID: "2b3600ee-a2f5-4b18-ac22-9c868136ce69") : object "kube-system"/"original-pull-secret" not registered Jul 20 21:02:50.768944 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:50.768709 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f09f6b9-6ed0-4165-bc40-706eb42f8f0d-metrics-certs podName:0f09f6b9-6ed0-4165-bc40-706eb42f8f0d nodeName:}" failed. No retries permitted until 2026-07-20 21:02:58.768699301 +0000 UTC m=+18.094402225 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0f09f6b9-6ed0-4165-bc40-706eb42f8f0d-metrics-certs") pod "network-metrics-daemon-t79t8" (UID: "0f09f6b9-6ed0-4165-bc40-706eb42f8f0d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 21:02:50.969903 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:50.969864 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzz2n\" (UniqueName: \"kubernetes.io/projected/f0fef424-8965-40f7-8b13-5a72773f050a-kube-api-access-hzz2n\") pod \"network-check-target-5chx7\" (UID: \"f0fef424-8965-40f7-8b13-5a72773f050a\") " pod="openshift-network-diagnostics/network-check-target-5chx7" Jul 20 21:02:50.970111 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:50.970035 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 21:02:50.970111 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:50.970054 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 21:02:50.970111 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:50.970067 2562 projected.go:196] Error preparing data for projected volume kube-api-access-hzz2n for pod openshift-network-diagnostics/network-check-target-5chx7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 21:02:50.970327 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:50.970129 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f0fef424-8965-40f7-8b13-5a72773f050a-kube-api-access-hzz2n podName:f0fef424-8965-40f7-8b13-5a72773f050a nodeName:}" failed. No retries permitted until 2026-07-20 21:02:58.970109537 +0000 UTC m=+18.295812464 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hzz2n" (UniqueName: "kubernetes.io/projected/f0fef424-8965-40f7-8b13-5a72773f050a-kube-api-access-hzz2n") pod "network-check-target-5chx7" (UID: "f0fef424-8965-40f7-8b13-5a72773f050a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 21:02:51.268318 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:51.268236 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2lqmn" Jul 20 21:02:51.268318 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:51.268281 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5chx7" Jul 20 21:02:51.268491 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:51.268360 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2lqmn" podUID="2b3600ee-a2f5-4b18-ac22-9c868136ce69" Jul 20 21:02:51.268575 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:51.268555 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t79t8" Jul 20 21:02:51.268689 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:51.268667 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5chx7" podUID="f0fef424-8965-40f7-8b13-5a72773f050a" Jul 20 21:02:51.268796 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:51.268761 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t79t8" podUID="0f09f6b9-6ed0-4165-bc40-706eb42f8f0d" Jul 20 21:02:53.264804 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:53.264646 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5chx7" Jul 20 21:02:53.264804 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:53.264661 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2lqmn" Jul 20 21:02:53.264804 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:53.264763 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5chx7" podUID="f0fef424-8965-40f7-8b13-5a72773f050a" Jul 20 21:02:53.265290 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:53.264831 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2lqmn" podUID="2b3600ee-a2f5-4b18-ac22-9c868136ce69" Jul 20 21:02:53.269159 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:53.265630 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t79t8" Jul 20 21:02:53.269159 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:53.265794 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t79t8" podUID="0f09f6b9-6ed0-4165-bc40-706eb42f8f0d" Jul 20 21:02:55.264012 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:55.263978 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t79t8" Jul 20 21:02:55.264012 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:55.264002 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2lqmn" Jul 20 21:02:55.264549 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:55.264002 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5chx7" Jul 20 21:02:55.264549 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:55.264107 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t79t8" podUID="0f09f6b9-6ed0-4165-bc40-706eb42f8f0d" Jul 20 21:02:55.264549 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:55.264218 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2lqmn" podUID="2b3600ee-a2f5-4b18-ac22-9c868136ce69" Jul 20 21:02:55.264549 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:55.264313 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5chx7" podUID="f0fef424-8965-40f7-8b13-5a72773f050a" Jul 20 21:02:57.264231 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:57.264194 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5chx7" Jul 20 21:02:57.264231 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:57.264233 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t79t8" Jul 20 21:02:57.264750 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:57.264201 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2lqmn" Jul 20 21:02:57.264750 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:57.264321 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5chx7" podUID="f0fef424-8965-40f7-8b13-5a72773f050a" Jul 20 21:02:57.264750 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:57.264390 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t79t8" podUID="0f09f6b9-6ed0-4165-bc40-706eb42f8f0d" Jul 20 21:02:57.264750 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:57.264501 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2lqmn" podUID="2b3600ee-a2f5-4b18-ac22-9c868136ce69" Jul 20 21:02:58.829564 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:58.829477 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b3600ee-a2f5-4b18-ac22-9c868136ce69-original-pull-secret\") pod \"global-pull-secret-syncer-2lqmn\" (UID: \"2b3600ee-a2f5-4b18-ac22-9c868136ce69\") " pod="kube-system/global-pull-secret-syncer-2lqmn" Jul 20 21:02:58.829564 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:58.829533 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f09f6b9-6ed0-4165-bc40-706eb42f8f0d-metrics-certs\") pod \"network-metrics-daemon-t79t8\" (UID: \"0f09f6b9-6ed0-4165-bc40-706eb42f8f0d\") " pod="openshift-multus/network-metrics-daemon-t79t8" Jul 20 21:02:58.830077 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:58.829641 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 21:02:58.830077 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:58.829698 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f09f6b9-6ed0-4165-bc40-706eb42f8f0d-metrics-certs podName:0f09f6b9-6ed0-4165-bc40-706eb42f8f0d nodeName:}" failed. No retries permitted until 2026-07-20 21:03:14.829682134 +0000 UTC m=+34.155385054 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0f09f6b9-6ed0-4165-bc40-706eb42f8f0d-metrics-certs") pod "network-metrics-daemon-t79t8" (UID: "0f09f6b9-6ed0-4165-bc40-706eb42f8f0d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 21:02:58.830077 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:58.829641 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 21:02:58.830077 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:58.829766 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b3600ee-a2f5-4b18-ac22-9c868136ce69-original-pull-secret podName:2b3600ee-a2f5-4b18-ac22-9c868136ce69 nodeName:}" failed. No retries permitted until 2026-07-20 21:03:14.829753889 +0000 UTC m=+34.155456823 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b3600ee-a2f5-4b18-ac22-9c868136ce69-original-pull-secret") pod "global-pull-secret-syncer-2lqmn" (UID: "2b3600ee-a2f5-4b18-ac22-9c868136ce69") : object "kube-system"/"original-pull-secret" not registered Jul 20 21:02:59.031295 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:59.031258 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzz2n\" (UniqueName: \"kubernetes.io/projected/f0fef424-8965-40f7-8b13-5a72773f050a-kube-api-access-hzz2n\") pod \"network-check-target-5chx7\" (UID: \"f0fef424-8965-40f7-8b13-5a72773f050a\") " pod="openshift-network-diagnostics/network-check-target-5chx7" Jul 20 21:02:59.031472 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:59.031403 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 21:02:59.031472 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:59.031426 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 21:02:59.031472 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:59.031437 2562 projected.go:196] Error preparing data for projected volume kube-api-access-hzz2n for pod openshift-network-diagnostics/network-check-target-5chx7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 21:02:59.031630 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:59.031494 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f0fef424-8965-40f7-8b13-5a72773f050a-kube-api-access-hzz2n podName:f0fef424-8965-40f7-8b13-5a72773f050a nodeName:}" failed. No retries permitted until 2026-07-20 21:03:15.031481316 +0000 UTC m=+34.357184237 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hzz2n" (UniqueName: "kubernetes.io/projected/f0fef424-8965-40f7-8b13-5a72773f050a-kube-api-access-hzz2n") pod "network-check-target-5chx7" (UID: "f0fef424-8965-40f7-8b13-5a72773f050a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 21:02:59.264171 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:59.264074 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2lqmn" Jul 20 21:02:59.264334 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:59.264076 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5chx7" Jul 20 21:02:59.264334 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:59.264207 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2lqmn" podUID="2b3600ee-a2f5-4b18-ac22-9c868136ce69" Jul 20 21:02:59.264334 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:59.264291 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5chx7" podUID="f0fef424-8965-40f7-8b13-5a72773f050a" Jul 20 21:02:59.264334 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:02:59.264077 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t79t8" Jul 20 21:02:59.264545 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:02:59.264400 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t79t8" podUID="0f09f6b9-6ed0-4165-bc40-706eb42f8f0d" Jul 20 21:03:01.265511 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:01.265464 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5chx7" Jul 20 21:03:01.265948 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:03:01.265560 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5chx7" podUID="f0fef424-8965-40f7-8b13-5a72773f050a" Jul 20 21:03:01.265948 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:01.265603 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t79t8" Jul 20 21:03:01.265948 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:01.265740 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2lqmn" Jul 20 21:03:01.265948 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:03:01.265737 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t79t8" podUID="0f09f6b9-6ed0-4165-bc40-706eb42f8f0d" Jul 20 21:03:01.265948 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:03:01.265831 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2lqmn" podUID="2b3600ee-a2f5-4b18-ac22-9c868136ce69" Jul 20 21:03:03.264996 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:03.264542 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5chx7" Jul 20 21:03:03.265779 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:03.264556 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2lqmn" Jul 20 21:03:03.265779 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:03:03.265057 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5chx7" podUID="f0fef424-8965-40f7-8b13-5a72773f050a" Jul 20 21:03:03.265779 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:03.264598 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t79t8" Jul 20 21:03:03.265779 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:03:03.265108 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2lqmn" podUID="2b3600ee-a2f5-4b18-ac22-9c868136ce69" Jul 20 21:03:03.265779 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:03:03.265214 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t79t8" podUID="0f09f6b9-6ed0-4165-bc40-706eb42f8f0d" Jul 20 21:03:03.363709 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:03.363673 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lgq8n" event={"ID":"ef68231f-0506-4450-9fcb-bc14af3b7d76","Type":"ContainerStarted","Data":"b22c9a13ab26da095e18de480d723cf9730de7599d1019e521ecf02041ebf735"} Jul 20 21:03:03.365089 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:03.365067 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7z98q" event={"ID":"e253d85e-aa94-494c-aaaf-daa7d9490119","Type":"ContainerStarted","Data":"4ed4b08f26d5bc23499ce1493164101a3414ea408d024c8641e92370f2beca09"} Jul 20 21:03:03.366404 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:03.366373 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdpbc" event={"ID":"68cad356-4b30-41ba-b68f-281c78eb24c1","Type":"ContainerStarted","Data":"91ca4c9cad37344efc5e6835533e7e393f81f549161638c9c0bf3a1532f42f29"} Jul 20 21:03:03.369082 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:03.369061 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" event={"ID":"356480cb-3559-4be3-a882-45d97dd6e493","Type":"ContainerStarted","Data":"9ea455491aa384693aaa4491d1d3b1c2a59cf36a7c712e6f226134aa2bb78945"} Jul 20 21:03:03.369202 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:03.369130 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" event={"ID":"356480cb-3559-4be3-a882-45d97dd6e493","Type":"ContainerStarted","Data":"1d87b2a391572f404e948b4444ee9906e7018df6303c29a21015def78a7423f6"} Jul 20 21:03:03.369265 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:03.369203 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" event={"ID":"356480cb-3559-4be3-a882-45d97dd6e493","Type":"ContainerStarted","Data":"a57f55ada5df9e5bbc7b849159cbe9295f4e3a4a19db2070951e823553046eeb"} Jul 20 21:03:03.369265 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:03.369249 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" event={"ID":"356480cb-3559-4be3-a882-45d97dd6e493","Type":"ContainerStarted","Data":"c5edb995dff6626f437d5e277a69e03ab5b0d679e3fd3dd37928493f3ce13737"} Jul 20 21:03:03.369364 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:03.369297 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" event={"ID":"356480cb-3559-4be3-a882-45d97dd6e493","Type":"ContainerStarted","Data":"f07f61175c865f051febc371ac25b3e4362e5d80e6012604cf8e136b9ccda6df"} Jul 20 21:03:03.369364 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:03.369344 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" event={"ID":"356480cb-3559-4be3-a882-45d97dd6e493","Type":"ContainerStarted","Data":"ec995352aad96322cde66a92e382d605e842af69049783ca5d67cccbdf44fe91"} Jul 20 21:03:03.370292 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:03.370268 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qcsfr" event={"ID":"dde8ddcb-7c86-4c9d-bd00-11129c8ca35e","Type":"ContainerStarted","Data":"d1445a888333e13746cd6e3ef8d1af040231a776b14553d46f39c55fec599096"} Jul 20 21:03:03.371885 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:03.371861 2562 generic.go:356] "Generic (PLEG): container finished" podID="c4188589-18dc-4bd1-9267-688596e53a7d" containerID="3c1357f6df7084f89b4a8743ccf57f0fbbe4b64c4278f360e9a342a8978da52d" exitCode=0 Jul 20 21:03:03.371967 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:03.371936 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mdg4m" event={"ID":"c4188589-18dc-4bd1-9267-688596e53a7d","Type":"ContainerDied","Data":"3c1357f6df7084f89b4a8743ccf57f0fbbe4b64c4278f360e9a342a8978da52d"} Jul 20 21:03:03.373959 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:03.373936 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5jlkf" event={"ID":"971c4fcc-cdb1-460e-9d0b-6eb07367f5c6","Type":"ContainerStarted","Data":"4a561b0d67b963f2218d16bb265cc3f4451f4b7202500aec17af7facde835a58"} Jul 20 21:03:03.375475 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:03.375454 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vz8z6" event={"ID":"a853a35b-650d-4ea4-81d3-d4c2ce54e9f1","Type":"ContainerStarted","Data":"06141ab722cef70b476de09ffb8ef50802088fdd39b33474f23c6405d59b478d"} Jul 20 21:03:03.379824 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:03.379774 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-32.ec2.internal" podStartSLOduration=21.379759903 podStartE2EDuration="21.379759903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 21:02:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:02:46.346458859 +0000 UTC m=+5.672161808" watchObservedRunningTime="2026-07-20 21:03:03.379759903 +0000 UTC m=+22.705462853" Jul 20 21:03:03.398054 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:03.397995 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vz8z6" podStartSLOduration=4.107389453 podStartE2EDuration="22.397979798s" totalImagesPullingTime="18.290590345s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 21:02:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:03:03.397970003 +0000 UTC m=+22.723672951" watchObservedRunningTime="2026-07-20 21:03:03.397979798 +0000 UTC m=+22.723682741" Jul 20 21:03:03.398422 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:03.398400 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lgq8n" podStartSLOduration=7.971650602 podStartE2EDuration="22.398390223s" totalImagesPullingTime="14.426739621s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 21:02:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:03:03.379579555 +0000 UTC m=+22.705282498" watchObservedRunningTime="2026-07-20 21:03:03.398390223 +0000 UTC m=+22.724093164" Jul 20 21:03:03.446234 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:03.446192 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7z98q" podStartSLOduration=4.216617601 podStartE2EDuration="22.446179162s" totalImagesPullingTime="18.229561561s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 21:02:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:03:03.445581409 +0000 UTC m=+22.771284353" watchObservedRunningTime="2026-07-20 21:03:03.446179162 +0000 UTC m=+22.771882105" Jul 20 21:03:03.466118 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:03.466077 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5jlkf" podStartSLOduration=8.036501277 podStartE2EDuration="22.466064007s" totalImagesPullingTime="14.42956273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 21:02:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:03:03.465947463 +0000 UTC m=+22.791650408" watchObservedRunningTime="2026-07-20 21:03:03.466064007 +0000 UTC m=+22.791766971" Jul 20 21:03:03.482533 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:03.482495 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-qcsfr" podStartSLOduration=8.054953465 podStartE2EDuration="22.48247983s" totalImagesPullingTime="14.427526365s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 21:02:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:03:03.482231058 +0000 UTC m=+22.807934000" watchObservedRunningTime="2026-07-20 21:03:03.48247983 +0000 UTC m=+22.808182771" Jul 20 21:03:03.574638 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:03.574615 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:03:03.670674 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:03.670649 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:03:03.798214 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:03.798194 2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 20 21:03:04.160262 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:04.160051 2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-20T21:03:03.798208863Z","UUID":"6e2e578b-9531-4455-9f96-77fcdbed2c24","Handler":null,"Name":"","Endpoint":""} Jul 20 21:03:04.163032 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:04.163010 2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 20 21:03:04.163184 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:04.163052 2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 20 21:03:04.379933 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:04.379903 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdpbc" event={"ID":"68cad356-4b30-41ba-b68f-281c78eb24c1","Type":"ContainerStarted","Data":"7bf7ba4f4129255806429ce788c54d2b633b7cd90a352473968bef88ad6e96df"} Jul 20 21:03:04.381439 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:04.381410 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-55bk8" event={"ID":"3d2247cc-c3ac-4a33-a206-a40a152d29fe","Type":"ContainerStarted","Data":"90fc70ead6636da78c114d1b8e50d1ffb6619a1fb6d4479c0f58c198e02867e0"} Jul 20 21:03:04.397266 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:04.397210 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-55bk8" podStartSLOduration=5.159031964 podStartE2EDuration="23.397197218s" totalImagesPullingTime="18.238165254s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 21:02:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:03:04.396641744 +0000 UTC m=+23.722344689" watchObservedRunningTime="2026-07-20 21:03:04.397197218 +0000 UTC m=+23.722900160" Jul 20 21:03:04.516426 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:04.516351 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:03:04.668153 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:04.668105 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:03:04.720942 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:04.720910 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:03:05.264119 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:05.264097 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t79t8" Jul 20 21:03:05.264256 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:05.264107 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5chx7" Jul 20 21:03:05.264256 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:05.264241 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2lqmn" Jul 20 21:03:05.264351 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:03:05.264237 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t79t8" podUID="0f09f6b9-6ed0-4165-bc40-706eb42f8f0d" Jul 20 21:03:05.264399 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:03:05.264362 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5chx7" podUID="f0fef424-8965-40f7-8b13-5a72773f050a" Jul 20 21:03:05.264466 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:03:05.264446 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2lqmn" podUID="2b3600ee-a2f5-4b18-ac22-9c868136ce69" Jul 20 21:03:05.387097 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:05.387019 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" event={"ID":"356480cb-3559-4be3-a882-45d97dd6e493","Type":"ContainerStarted","Data":"e99b244988e30a1643378185b0b9a34aea28b592ff2053939b79955acc586309"} Jul 20 21:03:06.391243 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:06.391029 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdpbc" event={"ID":"68cad356-4b30-41ba-b68f-281c78eb24c1","Type":"ContainerStarted","Data":"ccd98c532e71b27392f0c12619fd56a627b69326a83d05e8abeb697d30146496"} Jul 20 21:03:06.408908 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:06.408867 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdpbc" podStartSLOduration=4.284636865 podStartE2EDuration="25.408847803s" totalImagesPullingTime="21.124210938s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 21:02:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:03:06.407924446 +0000 UTC m=+25.733627389" watchObservedRunningTime="2026-07-20 21:03:06.408847803 +0000 UTC m=+25.734550746" Jul 20 21:03:07.264448 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:07.264411 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5chx7" Jul 20 21:03:07.264448 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:07.264432 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t79t8" Jul 20 21:03:07.264729 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:07.264411 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2lqmn" Jul 20 21:03:07.264729 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:03:07.264542 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5chx7" podUID="f0fef424-8965-40f7-8b13-5a72773f050a" Jul 20 21:03:07.264729 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:03:07.264621 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2lqmn" podUID="2b3600ee-a2f5-4b18-ac22-9c868136ce69" Jul 20 21:03:07.264882 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:03:07.264738 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t79t8" podUID="0f09f6b9-6ed0-4165-bc40-706eb42f8f0d" Jul 20 21:03:07.362294 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:07.362253 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5jlkf" Jul 20 21:03:07.363093 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:07.363073 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5jlkf" Jul 20 21:03:07.394032 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:07.393985 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5jlkf" Jul 20 21:03:07.394547 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:07.394529 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5jlkf" Jul 20 21:03:07.462433 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:07.462410 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:03:08.399209 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:08.398991 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" event={"ID":"356480cb-3559-4be3-a882-45d97dd6e493","Type":"ContainerStarted","Data":"490b156cc7f5c589823579407069c9e68e6cbba16a8d123f997b28269c0d725e"} Jul 20 21:03:08.400117 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:08.399272 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:03:08.400117 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:08.399323 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:03:08.400832 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:08.400808 2562 generic.go:356] "Generic (PLEG): container finished" podID="c4188589-18dc-4bd1-9267-688596e53a7d" containerID="75146f241ec9b7eb3deb11e7d256188474702d12e972a1921827339501908e0f" exitCode=0 Jul 20 21:03:08.400942 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:08.400878 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mdg4m" event={"ID":"c4188589-18dc-4bd1-9267-688596e53a7d","Type":"ContainerDied","Data":"75146f241ec9b7eb3deb11e7d256188474702d12e972a1921827339501908e0f"} Jul 20 21:03:08.413892 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:08.413873 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:03:08.426078 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:08.426043 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" podStartSLOduration=9.133242255 podStartE2EDuration="27.42602985s" totalImagesPullingTime="18.292787595s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 21:02:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:03:08.42558446 +0000 UTC m=+27.751287401" watchObservedRunningTime="2026-07-20 21:03:08.42602985 +0000 UTC m=+27.751732791" Jul 20 21:03:09.264732 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:09.264704 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2lqmn" Jul 20 21:03:09.264917 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:09.264704 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5chx7" Jul 20 21:03:09.264917 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:03:09.264826 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2lqmn" podUID="2b3600ee-a2f5-4b18-ac22-9c868136ce69" Jul 20 21:03:09.264917 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:09.264839 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t79t8" Jul 20 21:03:09.265024 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:03:09.264933 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5chx7" podUID="f0fef424-8965-40f7-8b13-5a72773f050a" Jul 20 21:03:09.265024 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:03:09.265000 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t79t8" podUID="0f09f6b9-6ed0-4165-bc40-706eb42f8f0d" Jul 20 21:03:09.403999 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:09.403978 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:03:09.426250 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:09.426218 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:03:09.465514 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:09.465487 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2lqmn"] Jul 20 21:03:09.465655 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:09.465638 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2lqmn" Jul 20 21:03:09.465769 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:03:09.465747 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2lqmn" podUID="2b3600ee-a2f5-4b18-ac22-9c868136ce69" Jul 20 21:03:09.468555 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:09.468517 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-t79t8"] Jul 20 21:03:09.468665 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:09.468644 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t79t8" Jul 20 21:03:09.468800 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:03:09.468778 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t79t8" podUID="0f09f6b9-6ed0-4165-bc40-706eb42f8f0d" Jul 20 21:03:09.469381 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:09.469360 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5chx7"] Jul 20 21:03:09.469471 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:09.469442 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5chx7" Jul 20 21:03:09.469539 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:03:09.469517 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5chx7" podUID="f0fef424-8965-40f7-8b13-5a72773f050a" Jul 20 21:03:09.633788 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:09.633717 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:03:09.698309 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:09.698277 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:03:10.407416 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:10.407384 2562 generic.go:356] "Generic (PLEG): container finished" podID="c4188589-18dc-4bd1-9267-688596e53a7d" containerID="928ae52cd8882b96dc48f5e4897741f467d0ca7936ce7bffc557fdbdff108ab0" exitCode=0 Jul 20 21:03:10.407826 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:10.407472 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mdg4m" event={"ID":"c4188589-18dc-4bd1-9267-688596e53a7d","Type":"ContainerDied","Data":"928ae52cd8882b96dc48f5e4897741f467d0ca7936ce7bffc557fdbdff108ab0"} Jul 20 21:03:11.265216 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:11.265023 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5chx7" Jul 20 21:03:11.265368 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:03:11.265282 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5chx7" podUID="f0fef424-8965-40f7-8b13-5a72773f050a" Jul 20 21:03:11.265368 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:11.265089 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2lqmn" Jul 20 21:03:11.265482 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:11.265119 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t79t8" Jul 20 21:03:11.265482 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:03:11.265418 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2lqmn" podUID="2b3600ee-a2f5-4b18-ac22-9c868136ce69" Jul 20 21:03:11.265572 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:03:11.265515 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t79t8" podUID="0f09f6b9-6ed0-4165-bc40-706eb42f8f0d" Jul 20 21:03:12.414460 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:12.414377 2562 generic.go:356] "Generic (PLEG): container finished" podID="c4188589-18dc-4bd1-9267-688596e53a7d" containerID="f9c1891f1240e8692f71d6d1ed17336072b38aaa1d122df3bf7fbe4658932d1b" exitCode=0 Jul 20 21:03:12.414460 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:12.414431 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mdg4m" event={"ID":"c4188589-18dc-4bd1-9267-688596e53a7d","Type":"ContainerDied","Data":"f9c1891f1240e8692f71d6d1ed17336072b38aaa1d122df3bf7fbe4658932d1b"} Jul 20 21:03:13.264224 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:13.264181 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2lqmn" Jul 20 21:03:13.264224 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:13.264218 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t79t8" Jul 20 21:03:13.264466 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:03:13.264311 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2lqmn" podUID="2b3600ee-a2f5-4b18-ac22-9c868136ce69" Jul 20 21:03:13.264466 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:13.264358 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5chx7" Jul 20 21:03:13.264466 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:03:13.264445 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5chx7" podUID="f0fef424-8965-40f7-8b13-5a72773f050a" Jul 20 21:03:13.264629 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:03:13.264534 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t79t8" podUID="0f09f6b9-6ed0-4165-bc40-706eb42f8f0d" Jul 20 21:03:14.849190 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:14.849156 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b3600ee-a2f5-4b18-ac22-9c868136ce69-original-pull-secret\") pod \"global-pull-secret-syncer-2lqmn\" (UID: \"2b3600ee-a2f5-4b18-ac22-9c868136ce69\") " pod="kube-system/global-pull-secret-syncer-2lqmn" Jul 20 21:03:14.849754 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:14.849200 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f09f6b9-6ed0-4165-bc40-706eb42f8f0d-metrics-certs\") pod \"network-metrics-daemon-t79t8\" (UID: \"0f09f6b9-6ed0-4165-bc40-706eb42f8f0d\") " pod="openshift-multus/network-metrics-daemon-t79t8" Jul 20 21:03:14.849754 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:03:14.849312 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 21:03:14.849754 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:03:14.849316 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 21:03:14.849754 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:03:14.849383 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f09f6b9-6ed0-4165-bc40-706eb42f8f0d-metrics-certs podName:0f09f6b9-6ed0-4165-bc40-706eb42f8f0d nodeName:}" failed. No retries permitted until 2026-07-20 21:03:46.849364489 +0000 UTC m=+66.175067414 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0f09f6b9-6ed0-4165-bc40-706eb42f8f0d-metrics-certs") pod "network-metrics-daemon-t79t8" (UID: "0f09f6b9-6ed0-4165-bc40-706eb42f8f0d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 21:03:14.849754 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:03:14.849401 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b3600ee-a2f5-4b18-ac22-9c868136ce69-original-pull-secret podName:2b3600ee-a2f5-4b18-ac22-9c868136ce69 nodeName:}" failed. No retries permitted until 2026-07-20 21:03:46.849393626 +0000 UTC m=+66.175096549 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b3600ee-a2f5-4b18-ac22-9c868136ce69-original-pull-secret") pod "global-pull-secret-syncer-2lqmn" (UID: "2b3600ee-a2f5-4b18-ac22-9c868136ce69") : object "kube-system"/"original-pull-secret" not registered Jul 20 21:03:15.051068 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:15.051030 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzz2n\" (UniqueName: \"kubernetes.io/projected/f0fef424-8965-40f7-8b13-5a72773f050a-kube-api-access-hzz2n\") pod \"network-check-target-5chx7\" (UID: \"f0fef424-8965-40f7-8b13-5a72773f050a\") " pod="openshift-network-diagnostics/network-check-target-5chx7" Jul 20 21:03:15.051255 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:03:15.051206 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 21:03:15.051255 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:03:15.051230 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 21:03:15.051255 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:03:15.051239 2562 projected.go:196] Error preparing data for projected volume kube-api-access-hzz2n for pod openshift-network-diagnostics/network-check-target-5chx7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 21:03:15.051358 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:03:15.051292 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f0fef424-8965-40f7-8b13-5a72773f050a-kube-api-access-hzz2n podName:f0fef424-8965-40f7-8b13-5a72773f050a nodeName:}" failed. No retries permitted until 2026-07-20 21:03:47.051277513 +0000 UTC m=+66.376980437 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hzz2n" (UniqueName: "kubernetes.io/projected/f0fef424-8965-40f7-8b13-5a72773f050a-kube-api-access-hzz2n") pod "network-check-target-5chx7" (UID: "f0fef424-8965-40f7-8b13-5a72773f050a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 21:03:15.264478 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:15.264440 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5chx7" Jul 20 21:03:15.264656 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:15.264444 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2lqmn" Jul 20 21:03:15.264656 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:03:15.264569 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5chx7" podUID="f0fef424-8965-40f7-8b13-5a72773f050a" Jul 20 21:03:15.264656 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:15.264441 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t79t8" Jul 20 21:03:15.264836 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:03:15.264653 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2lqmn" podUID="2b3600ee-a2f5-4b18-ac22-9c868136ce69" Jul 20 21:03:15.264836 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:03:15.264749 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t79t8" podUID="0f09f6b9-6ed0-4165-bc40-706eb42f8f0d" Jul 20 21:03:16.005772 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:16.005738 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-32.ec2.internal" event="NodeReady" Jul 20 21:03:16.006238 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:16.005898 2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 20 21:03:16.058253 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:16.058221 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zp575"] Jul 20 21:03:16.061185 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:16.061164 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jd2kv"] Jul 20 21:03:16.061323 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:16.061221 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zp575" Jul 20 21:03:16.062849 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:16.062830 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jd2kv" Jul 20 21:03:16.064443 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:16.064416 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 20 21:03:16.064827 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:16.064804 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-n29b9\"" Jul 20 21:03:16.064924 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:16.064804 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 20 21:03:16.065188 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:16.065168 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 20 21:03:16.065291 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:16.065251 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 20 21:03:16.065291 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:16.065278 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 20 21:03:16.065393 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:16.065342 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2wsgb\"" Jul 20 21:03:16.075058 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:16.075038 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jd2kv"] Jul 20 21:03:16.078473 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:16.078449 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zp575"] Jul 20 21:03:16.153898 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:16.153816 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:03:16.161288 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:16.161257 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2whtc\" (UniqueName: \"kubernetes.io/projected/7cc5fdb7-149c-47f7-805a-cadc7c5f2599-kube-api-access-2whtc\") pod \"dns-default-zp575\" (UID: \"7cc5fdb7-149c-47f7-805a-cadc7c5f2599\") " pod="openshift-dns/dns-default-zp575" Jul 20 21:03:16.161288 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:16.161298 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k82qv\" (UniqueName: \"kubernetes.io/projected/d1bb6ff3-c7d2-4be7-9e62-983c4f6bbff4-kube-api-access-k82qv\") pod \"ingress-canary-jd2kv\" (UID: \"d1bb6ff3-c7d2-4be7-9e62-983c4f6bbff4\") " pod="openshift-ingress-canary/ingress-canary-jd2kv" Jul 20 21:03:16.161500 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:16.161359 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d1bb6ff3-c7d2-4be7-9e62-983c4f6bbff4-cert\") pod \"ingress-canary-jd2kv\" (UID: \"d1bb6ff3-c7d2-4be7-9e62-983c4f6bbff4\") " pod="openshift-ingress-canary/ingress-canary-jd2kv" Jul 20 21:03:16.161500 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:16.161384 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cc5fdb7-149c-47f7-805a-cadc7c5f2599-metrics-tls\") pod \"dns-default-zp575\" (UID: \"7cc5fdb7-149c-47f7-805a-cadc7c5f2599\") " pod="openshift-dns/dns-default-zp575" Jul 20 21:03:16.161500 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:16.161477 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7cc5fdb7-149c-47f7-805a-cadc7c5f2599-config-volume\") pod \"dns-default-zp575\" (UID: \"7cc5fdb7-149c-47f7-805a-cadc7c5f2599\") " pod="openshift-dns/dns-default-zp575" Jul 20 21:03:16.161627 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:16.161519 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cc5fdb7-149c-47f7-805a-cadc7c5f2599-tmp-dir\") pod \"dns-default-zp575\" (UID: \"7cc5fdb7-149c-47f7-805a-cadc7c5f2599\") " pod="openshift-dns/dns-default-zp575" Jul 20 21:03:16.255602 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:16.255509 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:03:16.262383 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:16.262357 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d1bb6ff3-c7d2-4be7-9e62-983c4f6bbff4-cert\") pod \"ingress-canary-jd2kv\" (UID: \"d1bb6ff3-c7d2-4be7-9e62-983c4f6bbff4\") " pod="openshift-ingress-canary/ingress-canary-jd2kv" Jul 20 21:03:16.262533 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:16.262389 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cc5fdb7-149c-47f7-805a-cadc7c5f2599-metrics-tls\") pod \"dns-default-zp575\" (UID: \"7cc5fdb7-149c-47f7-805a-cadc7c5f2599\") " pod="openshift-dns/dns-default-zp575" Jul 20 21:03:16.262533 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:16.262428 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7cc5fdb7-149c-47f7-805a-cadc7c5f2599-config-volume\") pod \"dns-default-zp575\" (UID: \"7cc5fdb7-149c-47f7-805a-cadc7c5f2599\") " pod="openshift-dns/dns-default-zp575" Jul 20 21:03:16.262533 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:16.262454 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cc5fdb7-149c-47f7-805a-cadc7c5f2599-tmp-dir\") pod \"dns-default-zp575\" (UID: \"7cc5fdb7-149c-47f7-805a-cadc7c5f2599\") " pod="openshift-dns/dns-default-zp575" Jul 20 21:03:16.262533 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:16.262504 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2whtc\" (UniqueName: \"kubernetes.io/projected/7cc5fdb7-149c-47f7-805a-cadc7c5f2599-kube-api-access-2whtc\") pod \"dns-default-zp575\" (UID: \"7cc5fdb7-149c-47f7-805a-cadc7c5f2599\") " pod="openshift-dns/dns-default-zp575" Jul 20 21:03:16.262533 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:16.262528 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k82qv\" (UniqueName: \"kubernetes.io/projected/d1bb6ff3-c7d2-4be7-9e62-983c4f6bbff4-kube-api-access-k82qv\") pod \"ingress-canary-jd2kv\" (UID: \"d1bb6ff3-c7d2-4be7-9e62-983c4f6bbff4\") " pod="openshift-ingress-canary/ingress-canary-jd2kv" Jul 20 21:03:16.263117 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:03:16.262859 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 21:03:16.263117 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:03:16.262931 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cc5fdb7-149c-47f7-805a-cadc7c5f2599-metrics-tls podName:7cc5fdb7-149c-47f7-805a-cadc7c5f2599 nodeName:}" failed. No retries permitted until 2026-07-20 21:03:16.762913221 +0000 UTC m=+36.088616158 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7cc5fdb7-149c-47f7-805a-cadc7c5f2599-metrics-tls") pod "dns-default-zp575" (UID: "7cc5fdb7-149c-47f7-805a-cadc7c5f2599") : secret "dns-default-metrics-tls" not found Jul 20 21:03:16.263117 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:16.263083 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cc5fdb7-149c-47f7-805a-cadc7c5f2599-tmp-dir\") pod \"dns-default-zp575\" (UID: \"7cc5fdb7-149c-47f7-805a-cadc7c5f2599\") " pod="openshift-dns/dns-default-zp575" Jul 20 21:03:16.263358 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:16.263297 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7cc5fdb7-149c-47f7-805a-cadc7c5f2599-config-volume\") pod \"dns-default-zp575\" (UID: \"7cc5fdb7-149c-47f7-805a-cadc7c5f2599\") " pod="openshift-dns/dns-default-zp575" Jul 20 21:03:16.267764 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:16.267743 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d1bb6ff3-c7d2-4be7-9e62-983c4f6bbff4-cert\") pod \"ingress-canary-jd2kv\" (UID: \"d1bb6ff3-c7d2-4be7-9e62-983c4f6bbff4\") " pod="openshift-ingress-canary/ingress-canary-jd2kv" Jul 20 21:03:16.273519 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:16.273495 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k82qv\" (UniqueName: \"kubernetes.io/projected/d1bb6ff3-c7d2-4be7-9e62-983c4f6bbff4-kube-api-access-k82qv\") pod \"ingress-canary-jd2kv\" (UID: \"d1bb6ff3-c7d2-4be7-9e62-983c4f6bbff4\") " pod="openshift-ingress-canary/ingress-canary-jd2kv" Jul 20 21:03:16.273620 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:16.273530 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2whtc\" (UniqueName: \"kubernetes.io/projected/7cc5fdb7-149c-47f7-805a-cadc7c5f2599-kube-api-access-2whtc\") pod \"dns-default-zp575\" (UID: \"7cc5fdb7-149c-47f7-805a-cadc7c5f2599\") " pod="openshift-dns/dns-default-zp575" Jul 20 21:03:16.381844 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:16.381803 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jd2kv" Jul 20 21:03:16.637580 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:16.637400 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jd2kv"] Jul 20 21:03:16.640979 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:03:16.640944 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1bb6ff3_c7d2_4be7_9e62_983c4f6bbff4.slice/crio-f5feb997969970bc8e6fbe1ebcf158f743be1f2ae2713b5645d69b083b1aa5b9 WatchSource:0}: Error finding container f5feb997969970bc8e6fbe1ebcf158f743be1f2ae2713b5645d69b083b1aa5b9: Status 404 returned error can't find the container with id f5feb997969970bc8e6fbe1ebcf158f743be1f2ae2713b5645d69b083b1aa5b9 Jul 20 21:03:16.768389 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:16.768312 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cc5fdb7-149c-47f7-805a-cadc7c5f2599-metrics-tls\") pod \"dns-default-zp575\" (UID: \"7cc5fdb7-149c-47f7-805a-cadc7c5f2599\") " pod="openshift-dns/dns-default-zp575" Jul 20 21:03:16.768476 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:03:16.768452 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 21:03:16.768516 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:03:16.768507 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cc5fdb7-149c-47f7-805a-cadc7c5f2599-metrics-tls podName:7cc5fdb7-149c-47f7-805a-cadc7c5f2599 nodeName:}" failed. No retries permitted until 2026-07-20 21:03:17.768494251 +0000 UTC m=+37.094197176 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7cc5fdb7-149c-47f7-805a-cadc7c5f2599-metrics-tls") pod "dns-default-zp575" (UID: "7cc5fdb7-149c-47f7-805a-cadc7c5f2599") : secret "dns-default-metrics-tls" not found Jul 20 21:03:17.264844 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:17.264811 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t79t8" Jul 20 21:03:17.264844 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:17.264838 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2lqmn" Jul 20 21:03:17.265540 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:17.264811 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5chx7" Jul 20 21:03:17.268897 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:17.268594 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lfgsb\"" Jul 20 21:03:17.268897 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:17.268618 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 20 21:03:17.268897 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:17.268594 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 20 21:03:17.268897 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:17.268683 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hpjcj\"" Jul 20 21:03:17.268897 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:17.268727 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 20 21:03:17.268897 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:17.268619 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 20 21:03:17.431241 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:17.431211 2562 generic.go:356] "Generic (PLEG): container finished" podID="c4188589-18dc-4bd1-9267-688596e53a7d" containerID="82989c9b86e7721a26f82d5329ff6ad3f896980dfa4facd1a49dd32a3877c7e6" exitCode=0 Jul 20 21:03:17.431393 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:17.431288 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mdg4m" event={"ID":"c4188589-18dc-4bd1-9267-688596e53a7d","Type":"ContainerDied","Data":"82989c9b86e7721a26f82d5329ff6ad3f896980dfa4facd1a49dd32a3877c7e6"} Jul 20 21:03:17.432739 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:17.432498 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jd2kv" event={"ID":"d1bb6ff3-c7d2-4be7-9e62-983c4f6bbff4","Type":"ContainerStarted","Data":"f5feb997969970bc8e6fbe1ebcf158f743be1f2ae2713b5645d69b083b1aa5b9"} Jul 20 21:03:17.776066 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:17.775994 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cc5fdb7-149c-47f7-805a-cadc7c5f2599-metrics-tls\") pod \"dns-default-zp575\" (UID: \"7cc5fdb7-149c-47f7-805a-cadc7c5f2599\") " pod="openshift-dns/dns-default-zp575" Jul 20 21:03:17.776215 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:03:17.776155 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 21:03:17.776256 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:03:17.776230 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cc5fdb7-149c-47f7-805a-cadc7c5f2599-metrics-tls podName:7cc5fdb7-149c-47f7-805a-cadc7c5f2599 nodeName:}" failed. No retries permitted until 2026-07-20 21:03:19.776213371 +0000 UTC m=+39.101916293 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7cc5fdb7-149c-47f7-805a-cadc7c5f2599-metrics-tls") pod "dns-default-zp575" (UID: "7cc5fdb7-149c-47f7-805a-cadc7c5f2599") : secret "dns-default-metrics-tls" not found Jul 20 21:03:18.203455 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:18.203382 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:03:18.313409 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:18.313377 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:03:18.438449 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:18.438413 2562 generic.go:356] "Generic (PLEG): container finished" podID="c4188589-18dc-4bd1-9267-688596e53a7d" containerID="d55bfa1c1c490f3d44b5fa5e23029b9a4faa3de523a297e057d8a24f47217371" exitCode=0 Jul 20 21:03:18.438625 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:18.438464 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mdg4m" event={"ID":"c4188589-18dc-4bd1-9267-688596e53a7d","Type":"ContainerDied","Data":"d55bfa1c1c490f3d44b5fa5e23029b9a4faa3de523a297e057d8a24f47217371"} Jul 20 21:03:19.444542 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:19.444241 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mdg4m" event={"ID":"c4188589-18dc-4bd1-9267-688596e53a7d","Type":"ContainerStarted","Data":"dcd259f9a992a86590db50404f2e1be1e973fa144b28b1d9a7a1f207dac11469"} Jul 20 21:03:19.445641 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:19.445615 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jd2kv" event={"ID":"d1bb6ff3-c7d2-4be7-9e62-983c4f6bbff4","Type":"ContainerStarted","Data":"8ba690b35dd27ca69820629b8712fb9b699d1ae75a4a904a634b3290eb685f73"} Jul 20 21:03:19.468697 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:19.468645 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mdg4m" podStartSLOduration=9.892247113 podStartE2EDuration="38.468627122s" totalImagesPullingTime="28.576380009s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 21:02:41 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:03:19.467436012 +0000 UTC m=+38.793138954" watchObservedRunningTime="2026-07-20 21:03:19.468627122 +0000 UTC m=+38.794330065" Jul 20 21:03:19.483390 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:19.483338 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jd2kv" podStartSLOduration=1.262712362 podStartE2EDuration="3.483323761s" totalImagesPullingTime="2.220611399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 21:03:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:03:19.482823528 +0000 UTC m=+38.808526469" watchObservedRunningTime="2026-07-20 21:03:19.483323761 +0000 UTC m=+38.809026702" Jul 20 21:03:19.568428 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:19.568392 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:03:19.635032 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:19.634999 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:03:19.735194 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:19.735107 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:03:19.794181 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:19.794146 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cc5fdb7-149c-47f7-805a-cadc7c5f2599-metrics-tls\") pod \"dns-default-zp575\" (UID: \"7cc5fdb7-149c-47f7-805a-cadc7c5f2599\") " pod="openshift-dns/dns-default-zp575" Jul 20 21:03:19.794315 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:03:19.794261 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 21:03:19.794356 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:03:19.794313 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cc5fdb7-149c-47f7-805a-cadc7c5f2599-metrics-tls podName:7cc5fdb7-149c-47f7-805a-cadc7c5f2599 nodeName:}" failed. No retries permitted until 2026-07-20 21:03:23.794300137 +0000 UTC m=+43.120003057 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7cc5fdb7-149c-47f7-805a-cadc7c5f2599-metrics-tls") pod "dns-default-zp575" (UID: "7cc5fdb7-149c-47f7-805a-cadc7c5f2599") : secret "dns-default-metrics-tls" not found Jul 20 21:03:19.836531 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:19.836500 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:03:23.092856 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:23.092824 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:03:23.154744 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:23.154715 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:03:23.822567 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:23.822534 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cc5fdb7-149c-47f7-805a-cadc7c5f2599-metrics-tls\") pod \"dns-default-zp575\" (UID: \"7cc5fdb7-149c-47f7-805a-cadc7c5f2599\") " pod="openshift-dns/dns-default-zp575" Jul 20 21:03:23.822723 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:03:23.822696 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 21:03:23.822773 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:03:23.822762 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cc5fdb7-149c-47f7-805a-cadc7c5f2599-metrics-tls podName:7cc5fdb7-149c-47f7-805a-cadc7c5f2599 nodeName:}" failed. No retries permitted until 2026-07-20 21:03:31.822743018 +0000 UTC m=+51.148445938 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7cc5fdb7-149c-47f7-805a-cadc7c5f2599-metrics-tls") pod "dns-default-zp575" (UID: "7cc5fdb7-149c-47f7-805a-cadc7c5f2599") : secret "dns-default-metrics-tls" not found Jul 20 21:03:27.216166 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:27.216119 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:03:31.875323 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:31.875287 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cc5fdb7-149c-47f7-805a-cadc7c5f2599-metrics-tls\") pod \"dns-default-zp575\" (UID: \"7cc5fdb7-149c-47f7-805a-cadc7c5f2599\") " pod="openshift-dns/dns-default-zp575" Jul 20 21:03:31.875737 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:03:31.875438 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 21:03:31.875737 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:03:31.875500 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cc5fdb7-149c-47f7-805a-cadc7c5f2599-metrics-tls podName:7cc5fdb7-149c-47f7-805a-cadc7c5f2599 nodeName:}" failed. No retries permitted until 2026-07-20 21:03:47.875483116 +0000 UTC m=+67.201186055 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7cc5fdb7-149c-47f7-805a-cadc7c5f2599-metrics-tls") pod "dns-default-zp575" (UID: "7cc5fdb7-149c-47f7-805a-cadc7c5f2599") : secret "dns-default-metrics-tls" not found Jul 20 21:03:40.176622 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:40.176590 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:03:40.235919 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:40.235885 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:03:41.428717 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:41.428690 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9x7cd" Jul 20 21:03:41.543633 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:41.543606 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:03:41.606873 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:41.606842 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:03:41.667116 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:41.667085 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:03:46.880967 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:46.880909 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f09f6b9-6ed0-4165-bc40-706eb42f8f0d-metrics-certs\") pod \"network-metrics-daemon-t79t8\" (UID: \"0f09f6b9-6ed0-4165-bc40-706eb42f8f0d\") " pod="openshift-multus/network-metrics-daemon-t79t8" Jul 20 21:03:46.881476 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:46.881014 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b3600ee-a2f5-4b18-ac22-9c868136ce69-original-pull-secret\") pod \"global-pull-secret-syncer-2lqmn\" (UID: \"2b3600ee-a2f5-4b18-ac22-9c868136ce69\") " pod="kube-system/global-pull-secret-syncer-2lqmn" Jul 20 21:03:46.883877 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:46.883856 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 20 21:03:46.883933 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:46.883884 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 20 21:03:46.891837 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:03:46.891817 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 20 21:03:46.891885 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:03:46.891874 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f09f6b9-6ed0-4165-bc40-706eb42f8f0d-metrics-certs podName:0f09f6b9-6ed0-4165-bc40-706eb42f8f0d nodeName:}" failed. No retries permitted until 2026-07-20 21:04:50.891857688 +0000 UTC m=+130.217560609 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0f09f6b9-6ed0-4165-bc40-706eb42f8f0d-metrics-certs") pod "network-metrics-daemon-t79t8" (UID: "0f09f6b9-6ed0-4165-bc40-706eb42f8f0d") : secret "metrics-daemon-secret" not found Jul 20 21:03:46.895481 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:46.895465 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b3600ee-a2f5-4b18-ac22-9c868136ce69-original-pull-secret\") pod \"global-pull-secret-syncer-2lqmn\" (UID: \"2b3600ee-a2f5-4b18-ac22-9c868136ce69\") " pod="kube-system/global-pull-secret-syncer-2lqmn" Jul 20 21:03:46.993170 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:46.993107 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2lqmn" Jul 20 21:03:47.082830 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:47.082801 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzz2n\" (UniqueName: \"kubernetes.io/projected/f0fef424-8965-40f7-8b13-5a72773f050a-kube-api-access-hzz2n\") pod \"network-check-target-5chx7\" (UID: \"f0fef424-8965-40f7-8b13-5a72773f050a\") " pod="openshift-network-diagnostics/network-check-target-5chx7" Jul 20 21:03:47.085918 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:47.085896 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 20 21:03:47.095466 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:47.095446 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 20 21:03:47.105796 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:47.105771 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzz2n\" (UniqueName: \"kubernetes.io/projected/f0fef424-8965-40f7-8b13-5a72773f050a-kube-api-access-hzz2n\") pod \"network-check-target-5chx7\" (UID: \"f0fef424-8965-40f7-8b13-5a72773f050a\") " pod="openshift-network-diagnostics/network-check-target-5chx7" Jul 20 21:03:47.110307 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:47.110287 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2lqmn"] Jul 20 21:03:47.113817 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:03:47.113799 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b3600ee_a2f5_4b18_ac22_9c868136ce69.slice/crio-e3a3060f4c5ecfdfa3b90b6d67e32239ab0a410166082611ae383386179c358f WatchSource:0}: Error finding container e3a3060f4c5ecfdfa3b90b6d67e32239ab0a410166082611ae383386179c358f: Status 404 returned error can't find the container with id e3a3060f4c5ecfdfa3b90b6d67e32239ab0a410166082611ae383386179c358f Jul 20 21:03:47.290985 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:47.290964 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lfgsb\"" Jul 20 21:03:47.298082 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:47.298063 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5chx7" Jul 20 21:03:47.428948 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:47.428920 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5chx7"] Jul 20 21:03:47.432630 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:03:47.432607 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0fef424_8965_40f7_8b13_5a72773f050a.slice/crio-cab8dc5877acd177675bdebe5adb5140c1ec00e624da3d1e88ce688f434f4e04 WatchSource:0}: Error finding container cab8dc5877acd177675bdebe5adb5140c1ec00e624da3d1e88ce688f434f4e04: Status 404 returned error can't find the container with id cab8dc5877acd177675bdebe5adb5140c1ec00e624da3d1e88ce688f434f4e04 Jul 20 21:03:47.506849 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:47.506783 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5chx7" event={"ID":"f0fef424-8965-40f7-8b13-5a72773f050a","Type":"ContainerStarted","Data":"cab8dc5877acd177675bdebe5adb5140c1ec00e624da3d1e88ce688f434f4e04"} Jul 20 21:03:47.507967 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:47.507925 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2lqmn" event={"ID":"2b3600ee-a2f5-4b18-ac22-9c868136ce69","Type":"ContainerStarted","Data":"e3a3060f4c5ecfdfa3b90b6d67e32239ab0a410166082611ae383386179c358f"} Jul 20 21:03:47.888954 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:47.888916 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cc5fdb7-149c-47f7-805a-cadc7c5f2599-metrics-tls\") pod \"dns-default-zp575\" (UID: \"7cc5fdb7-149c-47f7-805a-cadc7c5f2599\") " pod="openshift-dns/dns-default-zp575" Jul 20 21:03:47.889428 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:03:47.889085 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 21:03:47.889428 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:03:47.889178 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cc5fdb7-149c-47f7-805a-cadc7c5f2599-metrics-tls podName:7cc5fdb7-149c-47f7-805a-cadc7c5f2599 nodeName:}" failed. No retries permitted until 2026-07-20 21:04:19.889156557 +0000 UTC m=+99.214859479 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7cc5fdb7-149c-47f7-805a-cadc7c5f2599-metrics-tls") pod "dns-default-zp575" (UID: "7cc5fdb7-149c-47f7-805a-cadc7c5f2599") : secret "dns-default-metrics-tls" not found Jul 20 21:03:52.524754 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:52.524713 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5chx7" event={"ID":"f0fef424-8965-40f7-8b13-5a72773f050a","Type":"ContainerStarted","Data":"5c1351f9cb6391fae1a37d41ea8d7ac4e4ea42b38f9d0126db80274569bf883e"} Jul 20 21:03:52.525279 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:52.524826 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5chx7" Jul 20 21:03:52.526193 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:52.526172 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2lqmn" event={"ID":"2b3600ee-a2f5-4b18-ac22-9c868136ce69","Type":"ContainerStarted","Data":"9ad45767ef1fdd6ca2c015f9bc59116f4502f9d8e6d82b933327c0ba4e60aee0"} Jul 20 21:03:52.543392 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:52.543345 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5chx7" podStartSLOduration=67.104077183 podStartE2EDuration="1m11.543332325s" totalImagesPullingTime="4.439255142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 21:02:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:03:52.542663381 +0000 UTC m=+71.868366323" watchObservedRunningTime="2026-07-20 21:03:52.543332325 +0000 UTC m=+71.869035264" Jul 20 21:03:52.557029 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:03:52.556983 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2lqmn" podStartSLOduration=66.798110776 podStartE2EDuration="1m11.556969596s" totalImagesPullingTime="4.75885882s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 21:02:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:03:52.556186692 +0000 UTC m=+71.881889626" watchObservedRunningTime="2026-07-20 21:03:52.556969596 +0000 UTC m=+71.882672539" Jul 20 21:04:09.021394 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:04:09.021360 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:04:19.910543 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:04:19.910503 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cc5fdb7-149c-47f7-805a-cadc7c5f2599-metrics-tls\") pod \"dns-default-zp575\" (UID: \"7cc5fdb7-149c-47f7-805a-cadc7c5f2599\") " pod="openshift-dns/dns-default-zp575" Jul 20 21:04:19.910936 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:04:19.910659 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 21:04:19.910936 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:04:19.910730 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cc5fdb7-149c-47f7-805a-cadc7c5f2599-metrics-tls podName:7cc5fdb7-149c-47f7-805a-cadc7c5f2599 nodeName:}" failed. No retries permitted until 2026-07-20 21:05:23.910710673 +0000 UTC m=+163.236413612 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7cc5fdb7-149c-47f7-805a-cadc7c5f2599-metrics-tls") pod "dns-default-zp575" (UID: "7cc5fdb7-149c-47f7-805a-cadc7c5f2599") : secret "dns-default-metrics-tls" not found Jul 20 21:04:23.531154 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:04:23.531112 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5chx7" Jul 20 21:04:50.918973 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:04:50.918921 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f09f6b9-6ed0-4165-bc40-706eb42f8f0d-metrics-certs\") pod \"network-metrics-daemon-t79t8\" (UID: \"0f09f6b9-6ed0-4165-bc40-706eb42f8f0d\") " pod="openshift-multus/network-metrics-daemon-t79t8" Jul 20 21:04:50.919552 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:04:50.919054 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 20 21:04:50.919552 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:04:50.919130 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f09f6b9-6ed0-4165-bc40-706eb42f8f0d-metrics-certs podName:0f09f6b9-6ed0-4165-bc40-706eb42f8f0d nodeName:}" failed. No retries permitted until 2026-07-20 21:06:52.919107905 +0000 UTC m=+252.244810839 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0f09f6b9-6ed0-4165-bc40-706eb42f8f0d-metrics-certs") pod "network-metrics-daemon-t79t8" (UID: "0f09f6b9-6ed0-4165-bc40-706eb42f8f0d") : secret "metrics-daemon-secret" not found Jul 20 21:05:17.225792 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:05:17.225741 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:05:19.074258 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:05:19.074221 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-zp575" podUID="7cc5fdb7-149c-47f7-805a-cadc7c5f2599" Jul 20 21:05:19.726035 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:05:19.726001 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zp575" Jul 20 21:05:20.281102 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:05:20.281059 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-t79t8" podUID="0f09f6b9-6ed0-4165-bc40-706eb42f8f0d" Jul 20 21:05:23.940959 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:05:23.940922 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cc5fdb7-149c-47f7-805a-cadc7c5f2599-metrics-tls\") pod \"dns-default-zp575\" (UID: \"7cc5fdb7-149c-47f7-805a-cadc7c5f2599\") " pod="openshift-dns/dns-default-zp575" Jul 20 21:05:23.941366 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:05:23.941056 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 21:05:23.941366 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:05:23.941116 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cc5fdb7-149c-47f7-805a-cadc7c5f2599-metrics-tls podName:7cc5fdb7-149c-47f7-805a-cadc7c5f2599 nodeName:}" failed. No retries permitted until 2026-07-20 21:07:25.9411002 +0000 UTC m=+285.266803125 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7cc5fdb7-149c-47f7-805a-cadc7c5f2599-metrics-tls") pod "dns-default-zp575" (UID: "7cc5fdb7-149c-47f7-805a-cadc7c5f2599") : secret "dns-default-metrics-tls" not found Jul 20 21:05:35.264470 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:05:35.264393 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t79t8" Jul 20 21:06:18.162634 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:18.162601 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:06:50.994637 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:50.994592 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-54chl"] Jul 20 21:06:50.996586 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:50.996568 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-mxgbd"] Jul 20 21:06:50.996739 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:50.996719 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-54chl" Jul 20 21:06:50.998846 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:50.998824 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-mxgbd" Jul 20 21:06:51.006514 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.006495 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 20 21:06:51.006514 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.006506 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 20 21:06:51.006633 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.006518 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 20 21:06:51.007275 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.007258 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 20 21:06:51.007372 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.007323 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 20 21:06:51.007452 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.007326 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 20 21:06:51.007452 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.007424 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-bqjxl\"" Jul 20 21:06:51.007452 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.007431 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 20 21:06:51.007587 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.007471 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-qp9k2\"" Jul 20 21:06:51.032164 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.032121 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5748dd95bb-qhfpf"] Jul 20 21:06:51.034509 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.034490 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5748dd95bb-qhfpf" Jul 20 21:06:51.038183 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.038163 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-54chl"] Jul 20 21:06:51.038942 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.038923 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-mxgbd"] Jul 20 21:06:51.044428 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.044411 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 20 21:06:51.044537 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.044519 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 20 21:06:51.044669 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.044645 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-24tzc\"" Jul 20 21:06:51.046357 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.046343 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 20 21:06:51.060005 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.059982 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 20 21:06:51.073407 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.073387 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5748dd95bb-qhfpf"] Jul 20 21:06:51.097311 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.097282 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8e63b84e-6cf9-467e-b114-9682be23c2b9-image-registry-private-configuration\") pod \"image-registry-5748dd95bb-qhfpf\" (UID: \"8e63b84e-6cf9-467e-b114-9682be23c2b9\") " pod="openshift-image-registry/image-registry-5748dd95bb-qhfpf" Jul 20 21:06:51.097425 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.097330 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpt4q\" (UniqueName: \"kubernetes.io/projected/da6da737-539a-4dbb-a8c1-994a4391da06-kube-api-access-qpt4q\") pod \"cluster-samples-operator-79b7c55f87-mxgbd\" (UID: \"da6da737-539a-4dbb-a8c1-994a4391da06\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-mxgbd" Jul 20 21:06:51.097425 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.097361 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a82a3265-df2c-4d0f-8aaf-d2434825fc10-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-54chl\" (UID: \"a82a3265-df2c-4d0f-8aaf-d2434825fc10\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-54chl" Jul 20 21:06:51.097425 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.097387 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfxjh\" (UniqueName: \"kubernetes.io/projected/8e63b84e-6cf9-467e-b114-9682be23c2b9-kube-api-access-mfxjh\") pod \"image-registry-5748dd95bb-qhfpf\" (UID: \"8e63b84e-6cf9-467e-b114-9682be23c2b9\") " pod="openshift-image-registry/image-registry-5748dd95bb-qhfpf" Jul 20 21:06:51.097425 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.097405 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88kv8\" (UniqueName: \"kubernetes.io/projected/a82a3265-df2c-4d0f-8aaf-d2434825fc10-kube-api-access-88kv8\") pod \"cluster-monitoring-operator-76cf8966f8-54chl\" (UID: \"a82a3265-df2c-4d0f-8aaf-d2434825fc10\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-54chl" Jul 20 21:06:51.097566 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.097462 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8e63b84e-6cf9-467e-b114-9682be23c2b9-trusted-ca\") pod \"image-registry-5748dd95bb-qhfpf\" (UID: \"8e63b84e-6cf9-467e-b114-9682be23c2b9\") " pod="openshift-image-registry/image-registry-5748dd95bb-qhfpf" Jul 20 21:06:51.097566 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.097490 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8e63b84e-6cf9-467e-b114-9682be23c2b9-installation-pull-secrets\") pod \"image-registry-5748dd95bb-qhfpf\" (UID: \"8e63b84e-6cf9-467e-b114-9682be23c2b9\") " pod="openshift-image-registry/image-registry-5748dd95bb-qhfpf" Jul 20 21:06:51.097566 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.097507 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8e63b84e-6cf9-467e-b114-9682be23c2b9-bound-sa-token\") pod \"image-registry-5748dd95bb-qhfpf\" (UID: \"8e63b84e-6cf9-467e-b114-9682be23c2b9\") " pod="openshift-image-registry/image-registry-5748dd95bb-qhfpf" Jul 20 21:06:51.097566 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.097528 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/da6da737-539a-4dbb-a8c1-994a4391da06-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-mxgbd\" (UID: \"da6da737-539a-4dbb-a8c1-994a4391da06\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-mxgbd" Jul 20 21:06:51.097566 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.097542 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8e63b84e-6cf9-467e-b114-9682be23c2b9-registry-tls\") pod \"image-registry-5748dd95bb-qhfpf\" (UID: \"8e63b84e-6cf9-467e-b114-9682be23c2b9\") " pod="openshift-image-registry/image-registry-5748dd95bb-qhfpf" Jul 20 21:06:51.097783 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.097635 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8e63b84e-6cf9-467e-b114-9682be23c2b9-ca-trust-extracted\") pod \"image-registry-5748dd95bb-qhfpf\" (UID: \"8e63b84e-6cf9-467e-b114-9682be23c2b9\") " pod="openshift-image-registry/image-registry-5748dd95bb-qhfpf" Jul 20 21:06:51.097783 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.097675 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8e63b84e-6cf9-467e-b114-9682be23c2b9-registry-certificates\") pod \"image-registry-5748dd95bb-qhfpf\" (UID: \"8e63b84e-6cf9-467e-b114-9682be23c2b9\") " pod="openshift-image-registry/image-registry-5748dd95bb-qhfpf" Jul 20 21:06:51.097783 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.097705 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a82a3265-df2c-4d0f-8aaf-d2434825fc10-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-54chl\" (UID: \"a82a3265-df2c-4d0f-8aaf-d2434825fc10\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-54chl" Jul 20 21:06:51.128504 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.128473 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-lv5br"] Jul 20 21:06:51.131021 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.130998 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-lv5br" Jul 20 21:06:51.131241 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.131212 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-nft7r"] Jul 20 21:06:51.132879 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.132864 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-nft7r" Jul 20 21:06:51.134734 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.134717 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-ztx8d\"" Jul 20 21:06:51.138608 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.138592 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 20 21:06:51.138862 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.138844 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 20 21:06:51.143471 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.143452 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 20 21:06:51.143908 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.143895 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 20 21:06:51.145710 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.145692 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-r8knq\"" Jul 20 21:06:51.145778 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.145692 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 20 21:06:51.145778 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.145692 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 20 21:06:51.159677 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.159654 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-nft7r"] Jul 20 21:06:51.167006 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.166983 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-lv5br"] Jul 20 21:06:51.198496 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.198472 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxdbf\" (UniqueName: \"kubernetes.io/projected/5ef2d9b0-d68d-417e-bc1e-3fcd873aa850-kube-api-access-jxdbf\") pod \"volume-data-source-validator-78cf4c6f6-nft7r\" (UID: \"5ef2d9b0-d68d-417e-bc1e-3fcd873aa850\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-nft7r" Jul 20 21:06:51.198647 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.198509 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8e63b84e-6cf9-467e-b114-9682be23c2b9-registry-certificates\") pod \"image-registry-5748dd95bb-qhfpf\" (UID: \"8e63b84e-6cf9-467e-b114-9682be23c2b9\") " pod="openshift-image-registry/image-registry-5748dd95bb-qhfpf" Jul 20 21:06:51.198647 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.198528 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a82a3265-df2c-4d0f-8aaf-d2434825fc10-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-54chl\" (UID: \"a82a3265-df2c-4d0f-8aaf-d2434825fc10\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-54chl" Jul 20 21:06:51.198647 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.198543 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a002a221-d9f5-42b1-b07b-4b6e946cf5c5-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-lv5br\" (UID: \"a002a221-d9f5-42b1-b07b-4b6e946cf5c5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-lv5br" Jul 20 21:06:51.198647 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.198565 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8e63b84e-6cf9-467e-b114-9682be23c2b9-image-registry-private-configuration\") pod \"image-registry-5748dd95bb-qhfpf\" (UID: \"8e63b84e-6cf9-467e-b114-9682be23c2b9\") " pod="openshift-image-registry/image-registry-5748dd95bb-qhfpf" Jul 20 21:06:51.198647 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.198599 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpt4q\" (UniqueName: \"kubernetes.io/projected/da6da737-539a-4dbb-a8c1-994a4391da06-kube-api-access-qpt4q\") pod \"cluster-samples-operator-79b7c55f87-mxgbd\" (UID: \"da6da737-539a-4dbb-a8c1-994a4391da06\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-mxgbd" Jul 20 21:06:51.198647 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.198617 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a82a3265-df2c-4d0f-8aaf-d2434825fc10-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-54chl\" (UID: \"a82a3265-df2c-4d0f-8aaf-d2434825fc10\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-54chl" Jul 20 21:06:51.198647 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.198630 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a002a221-d9f5-42b1-b07b-4b6e946cf5c5-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-lv5br\" (UID: \"a002a221-d9f5-42b1-b07b-4b6e946cf5c5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-lv5br" Jul 20 21:06:51.198981 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.198674 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfxjh\" (UniqueName: \"kubernetes.io/projected/8e63b84e-6cf9-467e-b114-9682be23c2b9-kube-api-access-mfxjh\") pod \"image-registry-5748dd95bb-qhfpf\" (UID: \"8e63b84e-6cf9-467e-b114-9682be23c2b9\") " pod="openshift-image-registry/image-registry-5748dd95bb-qhfpf" Jul 20 21:06:51.198981 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.198714 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88kv8\" (UniqueName: \"kubernetes.io/projected/a82a3265-df2c-4d0f-8aaf-d2434825fc10-kube-api-access-88kv8\") pod \"cluster-monitoring-operator-76cf8966f8-54chl\" (UID: \"a82a3265-df2c-4d0f-8aaf-d2434825fc10\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-54chl" Jul 20 21:06:51.198981 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:51.198737 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 20 21:06:51.198981 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.198743 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2zcz\" (UniqueName: \"kubernetes.io/projected/a002a221-d9f5-42b1-b07b-4b6e946cf5c5-kube-api-access-b2zcz\") pod \"kube-storage-version-migrator-operator-7658d9b749-lv5br\" (UID: \"a002a221-d9f5-42b1-b07b-4b6e946cf5c5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-lv5br" Jul 20 21:06:51.198981 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.198792 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8e63b84e-6cf9-467e-b114-9682be23c2b9-trusted-ca\") pod \"image-registry-5748dd95bb-qhfpf\" (UID: \"8e63b84e-6cf9-467e-b114-9682be23c2b9\") " pod="openshift-image-registry/image-registry-5748dd95bb-qhfpf" Jul 20 21:06:51.198981 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:51.198812 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a82a3265-df2c-4d0f-8aaf-d2434825fc10-cluster-monitoring-operator-tls podName:a82a3265-df2c-4d0f-8aaf-d2434825fc10 nodeName:}" failed. No retries permitted until 2026-07-20 21:06:51.698793601 +0000 UTC m=+251.024496530 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a82a3265-df2c-4d0f-8aaf-d2434825fc10-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-54chl" (UID: "a82a3265-df2c-4d0f-8aaf-d2434825fc10") : secret "cluster-monitoring-operator-tls" not found Jul 20 21:06:51.198981 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.198910 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8e63b84e-6cf9-467e-b114-9682be23c2b9-installation-pull-secrets\") pod \"image-registry-5748dd95bb-qhfpf\" (UID: \"8e63b84e-6cf9-467e-b114-9682be23c2b9\") " pod="openshift-image-registry/image-registry-5748dd95bb-qhfpf" Jul 20 21:06:51.198981 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.198935 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8e63b84e-6cf9-467e-b114-9682be23c2b9-bound-sa-token\") pod \"image-registry-5748dd95bb-qhfpf\" (UID: \"8e63b84e-6cf9-467e-b114-9682be23c2b9\") " pod="openshift-image-registry/image-registry-5748dd95bb-qhfpf" Jul 20 21:06:51.198981 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.198971 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/da6da737-539a-4dbb-a8c1-994a4391da06-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-mxgbd\" (UID: \"da6da737-539a-4dbb-a8c1-994a4391da06\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-mxgbd" Jul 20 21:06:51.199495 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.198995 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8e63b84e-6cf9-467e-b114-9682be23c2b9-registry-tls\") pod \"image-registry-5748dd95bb-qhfpf\" (UID: \"8e63b84e-6cf9-467e-b114-9682be23c2b9\") " pod="openshift-image-registry/image-registry-5748dd95bb-qhfpf" Jul 20 21:06:51.199495 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.199030 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8e63b84e-6cf9-467e-b114-9682be23c2b9-ca-trust-extracted\") pod \"image-registry-5748dd95bb-qhfpf\" (UID: \"8e63b84e-6cf9-467e-b114-9682be23c2b9\") " pod="openshift-image-registry/image-registry-5748dd95bb-qhfpf" Jul 20 21:06:51.199495 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:51.199098 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 20 21:06:51.199495 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:51.199184 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da6da737-539a-4dbb-a8c1-994a4391da06-samples-operator-tls podName:da6da737-539a-4dbb-a8c1-994a4391da06 nodeName:}" failed. No retries permitted until 2026-07-20 21:06:51.69916673 +0000 UTC m=+251.024869670 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/da6da737-539a-4dbb-a8c1-994a4391da06-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-mxgbd" (UID: "da6da737-539a-4dbb-a8c1-994a4391da06") : secret "samples-operator-tls" not found Jul 20 21:06:51.199495 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.199288 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8e63b84e-6cf9-467e-b114-9682be23c2b9-registry-certificates\") pod \"image-registry-5748dd95bb-qhfpf\" (UID: \"8e63b84e-6cf9-467e-b114-9682be23c2b9\") " pod="openshift-image-registry/image-registry-5748dd95bb-qhfpf" Jul 20 21:06:51.199495 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.199338 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a82a3265-df2c-4d0f-8aaf-d2434825fc10-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-54chl\" (UID: \"a82a3265-df2c-4d0f-8aaf-d2434825fc10\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-54chl" Jul 20 21:06:51.199495 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:51.199380 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 20 21:06:51.199495 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.199386 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8e63b84e-6cf9-467e-b114-9682be23c2b9-ca-trust-extracted\") pod \"image-registry-5748dd95bb-qhfpf\" (UID: \"8e63b84e-6cf9-467e-b114-9682be23c2b9\") " pod="openshift-image-registry/image-registry-5748dd95bb-qhfpf" Jul 20 21:06:51.199495 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:51.199392 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5748dd95bb-qhfpf: secret "image-registry-tls" not found Jul 20 21:06:51.199779 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:51.199525 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e63b84e-6cf9-467e-b114-9682be23c2b9-registry-tls podName:8e63b84e-6cf9-467e-b114-9682be23c2b9 nodeName:}" failed. No retries permitted until 2026-07-20 21:06:51.699510237 +0000 UTC m=+251.025213175 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8e63b84e-6cf9-467e-b114-9682be23c2b9-registry-tls") pod "image-registry-5748dd95bb-qhfpf" (UID: "8e63b84e-6cf9-467e-b114-9682be23c2b9") : secret "image-registry-tls" not found Jul 20 21:06:51.199845 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.199820 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8e63b84e-6cf9-467e-b114-9682be23c2b9-trusted-ca\") pod \"image-registry-5748dd95bb-qhfpf\" (UID: \"8e63b84e-6cf9-467e-b114-9682be23c2b9\") " pod="openshift-image-registry/image-registry-5748dd95bb-qhfpf" Jul 20 21:06:51.201304 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.201279 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8e63b84e-6cf9-467e-b114-9682be23c2b9-image-registry-private-configuration\") pod \"image-registry-5748dd95bb-qhfpf\" (UID: \"8e63b84e-6cf9-467e-b114-9682be23c2b9\") " pod="openshift-image-registry/image-registry-5748dd95bb-qhfpf" Jul 20 21:06:51.201382 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.201319 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8e63b84e-6cf9-467e-b114-9682be23c2b9-installation-pull-secrets\") pod \"image-registry-5748dd95bb-qhfpf\" (UID: \"8e63b84e-6cf9-467e-b114-9682be23c2b9\") " pod="openshift-image-registry/image-registry-5748dd95bb-qhfpf" Jul 20 21:06:51.208451 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.208423 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfxjh\" (UniqueName: \"kubernetes.io/projected/8e63b84e-6cf9-467e-b114-9682be23c2b9-kube-api-access-mfxjh\") pod \"image-registry-5748dd95bb-qhfpf\" (UID: \"8e63b84e-6cf9-467e-b114-9682be23c2b9\") " pod="openshift-image-registry/image-registry-5748dd95bb-qhfpf" Jul 20 21:06:51.213386 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.213368 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8e63b84e-6cf9-467e-b114-9682be23c2b9-bound-sa-token\") pod \"image-registry-5748dd95bb-qhfpf\" (UID: \"8e63b84e-6cf9-467e-b114-9682be23c2b9\") " pod="openshift-image-registry/image-registry-5748dd95bb-qhfpf" Jul 20 21:06:51.213463 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.213400 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88kv8\" (UniqueName: \"kubernetes.io/projected/a82a3265-df2c-4d0f-8aaf-d2434825fc10-kube-api-access-88kv8\") pod \"cluster-monitoring-operator-76cf8966f8-54chl\" (UID: \"a82a3265-df2c-4d0f-8aaf-d2434825fc10\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-54chl" Jul 20 21:06:51.215334 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.215312 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpt4q\" (UniqueName: \"kubernetes.io/projected/da6da737-539a-4dbb-a8c1-994a4391da06-kube-api-access-qpt4q\") pod \"cluster-samples-operator-79b7c55f87-mxgbd\" (UID: \"da6da737-539a-4dbb-a8c1-994a4391da06\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-mxgbd" Jul 20 21:06:51.299712 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.299674 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a002a221-d9f5-42b1-b07b-4b6e946cf5c5-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-lv5br\" (UID: \"a002a221-d9f5-42b1-b07b-4b6e946cf5c5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-lv5br" Jul 20 21:06:51.299899 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.299737 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a002a221-d9f5-42b1-b07b-4b6e946cf5c5-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-lv5br\" (UID: \"a002a221-d9f5-42b1-b07b-4b6e946cf5c5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-lv5br" Jul 20 21:06:51.299899 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.299757 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2zcz\" (UniqueName: \"kubernetes.io/projected/a002a221-d9f5-42b1-b07b-4b6e946cf5c5-kube-api-access-b2zcz\") pod \"kube-storage-version-migrator-operator-7658d9b749-lv5br\" (UID: \"a002a221-d9f5-42b1-b07b-4b6e946cf5c5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-lv5br" Jul 20 21:06:51.299899 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.299813 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxdbf\" (UniqueName: \"kubernetes.io/projected/5ef2d9b0-d68d-417e-bc1e-3fcd873aa850-kube-api-access-jxdbf\") pod \"volume-data-source-validator-78cf4c6f6-nft7r\" (UID: \"5ef2d9b0-d68d-417e-bc1e-3fcd873aa850\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-nft7r" Jul 20 21:06:51.300278 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.300251 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a002a221-d9f5-42b1-b07b-4b6e946cf5c5-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-lv5br\" (UID: \"a002a221-d9f5-42b1-b07b-4b6e946cf5c5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-lv5br" Jul 20 21:06:51.301896 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.301874 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a002a221-d9f5-42b1-b07b-4b6e946cf5c5-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-lv5br\" (UID: \"a002a221-d9f5-42b1-b07b-4b6e946cf5c5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-lv5br" Jul 20 21:06:51.311299 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.311268 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:06:51.313378 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.313362 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxdbf\" (UniqueName: \"kubernetes.io/projected/5ef2d9b0-d68d-417e-bc1e-3fcd873aa850-kube-api-access-jxdbf\") pod \"volume-data-source-validator-78cf4c6f6-nft7r\" (UID: \"5ef2d9b0-d68d-417e-bc1e-3fcd873aa850\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-nft7r" Jul 20 21:06:51.317638 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.317620 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2zcz\" (UniqueName: \"kubernetes.io/projected/a002a221-d9f5-42b1-b07b-4b6e946cf5c5-kube-api-access-b2zcz\") pod \"kube-storage-version-migrator-operator-7658d9b749-lv5br\" (UID: \"a002a221-d9f5-42b1-b07b-4b6e946cf5c5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-lv5br" Jul 20 21:06:51.416063 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.416025 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:06:51.440963 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.440939 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-lv5br" Jul 20 21:06:51.445627 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.445608 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-nft7r" Jul 20 21:06:51.581023 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.580957 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-lv5br"] Jul 20 21:06:51.583646 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:06:51.583624 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda002a221_d9f5_42b1_b07b_4b6e946cf5c5.slice/crio-5368c7fa58bc3218bcf1c1b22428e067b4086cbe75becb150421f2b36d0eb6b8 WatchSource:0}: Error finding container 5368c7fa58bc3218bcf1c1b22428e067b4086cbe75becb150421f2b36d0eb6b8: Status 404 returned error can't find the container with id 5368c7fa58bc3218bcf1c1b22428e067b4086cbe75becb150421f2b36d0eb6b8 Jul 20 21:06:51.593729 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.593706 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-nft7r"] Jul 20 21:06:51.596344 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:06:51.596323 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ef2d9b0_d68d_417e_bc1e_3fcd873aa850.slice/crio-7a94daf991507b17723ae9386618d3c4518cac23dbab5f12bb33263eafb9bdc1 WatchSource:0}: Error finding container 7a94daf991507b17723ae9386618d3c4518cac23dbab5f12bb33263eafb9bdc1: Status 404 returned error can't find the container with id 7a94daf991507b17723ae9386618d3c4518cac23dbab5f12bb33263eafb9bdc1 Jul 20 21:06:51.702049 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.702011 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a82a3265-df2c-4d0f-8aaf-d2434825fc10-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-54chl\" (UID: \"a82a3265-df2c-4d0f-8aaf-d2434825fc10\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-54chl" Jul 20 21:06:51.702214 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.702063 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/da6da737-539a-4dbb-a8c1-994a4391da06-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-mxgbd\" (UID: \"da6da737-539a-4dbb-a8c1-994a4391da06\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-mxgbd" Jul 20 21:06:51.702214 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.702080 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8e63b84e-6cf9-467e-b114-9682be23c2b9-registry-tls\") pod \"image-registry-5748dd95bb-qhfpf\" (UID: \"8e63b84e-6cf9-467e-b114-9682be23c2b9\") " pod="openshift-image-registry/image-registry-5748dd95bb-qhfpf" Jul 20 21:06:51.702214 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:51.702169 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 20 21:06:51.702214 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:51.702183 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 20 21:06:51.702214 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:51.702194 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5748dd95bb-qhfpf: secret "image-registry-tls" not found Jul 20 21:06:51.702389 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:51.702169 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 20 21:06:51.702389 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:51.702229 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a82a3265-df2c-4d0f-8aaf-d2434825fc10-cluster-monitoring-operator-tls podName:a82a3265-df2c-4d0f-8aaf-d2434825fc10 nodeName:}" failed. No retries permitted until 2026-07-20 21:06:52.70221474 +0000 UTC m=+252.027917664 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a82a3265-df2c-4d0f-8aaf-d2434825fc10-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-54chl" (UID: "a82a3265-df2c-4d0f-8aaf-d2434825fc10") : secret "cluster-monitoring-operator-tls" not found Jul 20 21:06:51.702389 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:51.702244 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e63b84e-6cf9-467e-b114-9682be23c2b9-registry-tls podName:8e63b84e-6cf9-467e-b114-9682be23c2b9 nodeName:}" failed. No retries permitted until 2026-07-20 21:06:52.702237528 +0000 UTC m=+252.027940448 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8e63b84e-6cf9-467e-b114-9682be23c2b9-registry-tls") pod "image-registry-5748dd95bb-qhfpf" (UID: "8e63b84e-6cf9-467e-b114-9682be23c2b9") : secret "image-registry-tls" not found Jul 20 21:06:51.702389 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:51.702253 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da6da737-539a-4dbb-a8c1-994a4391da06-samples-operator-tls podName:da6da737-539a-4dbb-a8c1-994a4391da06 nodeName:}" failed. No retries permitted until 2026-07-20 21:06:52.702248457 +0000 UTC m=+252.027951377 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/da6da737-539a-4dbb-a8c1-994a4391da06-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-mxgbd" (UID: "da6da737-539a-4dbb-a8c1-994a4391da06") : secret "samples-operator-tls" not found Jul 20 21:06:51.936701 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.936617 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-nft7r" event={"ID":"5ef2d9b0-d68d-417e-bc1e-3fcd873aa850","Type":"ContainerStarted","Data":"7a94daf991507b17723ae9386618d3c4518cac23dbab5f12bb33263eafb9bdc1"} Jul 20 21:06:51.937635 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:51.937612 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-lv5br" event={"ID":"a002a221-d9f5-42b1-b07b-4b6e946cf5c5","Type":"ContainerStarted","Data":"5368c7fa58bc3218bcf1c1b22428e067b4086cbe75becb150421f2b36d0eb6b8"} Jul 20 21:06:52.710724 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:52.710686 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a82a3265-df2c-4d0f-8aaf-d2434825fc10-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-54chl\" (UID: \"a82a3265-df2c-4d0f-8aaf-d2434825fc10\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-54chl" Jul 20 21:06:52.711210 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:52.710746 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/da6da737-539a-4dbb-a8c1-994a4391da06-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-mxgbd\" (UID: \"da6da737-539a-4dbb-a8c1-994a4391da06\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-mxgbd" Jul 20 21:06:52.711210 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:52.710764 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8e63b84e-6cf9-467e-b114-9682be23c2b9-registry-tls\") pod \"image-registry-5748dd95bb-qhfpf\" (UID: \"8e63b84e-6cf9-467e-b114-9682be23c2b9\") " pod="openshift-image-registry/image-registry-5748dd95bb-qhfpf" Jul 20 21:06:52.711210 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:52.710844 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 20 21:06:52.711210 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:52.710915 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a82a3265-df2c-4d0f-8aaf-d2434825fc10-cluster-monitoring-operator-tls podName:a82a3265-df2c-4d0f-8aaf-d2434825fc10 nodeName:}" failed. No retries permitted until 2026-07-20 21:06:54.7109 +0000 UTC m=+254.036602920 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a82a3265-df2c-4d0f-8aaf-d2434825fc10-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-54chl" (UID: "a82a3265-df2c-4d0f-8aaf-d2434825fc10") : secret "cluster-monitoring-operator-tls" not found Jul 20 21:06:52.711210 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:52.710914 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 20 21:06:52.711210 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:52.710859 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 20 21:06:52.711210 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:52.710946 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5748dd95bb-qhfpf: secret "image-registry-tls" not found Jul 20 21:06:52.711210 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:52.710973 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da6da737-539a-4dbb-a8c1-994a4391da06-samples-operator-tls podName:da6da737-539a-4dbb-a8c1-994a4391da06 nodeName:}" failed. No retries permitted until 2026-07-20 21:06:54.710956669 +0000 UTC m=+254.036659592 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/da6da737-539a-4dbb-a8c1-994a4391da06-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-mxgbd" (UID: "da6da737-539a-4dbb-a8c1-994a4391da06") : secret "samples-operator-tls" not found Jul 20 21:06:52.711210 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:52.710998 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e63b84e-6cf9-467e-b114-9682be23c2b9-registry-tls podName:8e63b84e-6cf9-467e-b114-9682be23c2b9 nodeName:}" failed. No retries permitted until 2026-07-20 21:06:54.710985107 +0000 UTC m=+254.036688031 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8e63b84e-6cf9-467e-b114-9682be23c2b9-registry-tls") pod "image-registry-5748dd95bb-qhfpf" (UID: "8e63b84e-6cf9-467e-b114-9682be23c2b9") : secret "image-registry-tls" not found Jul 20 21:06:52.941691 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:52.941657 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-nft7r" event={"ID":"5ef2d9b0-d68d-417e-bc1e-3fcd873aa850","Type":"ContainerStarted","Data":"58ae406ff06e78bcfcfa9aa0adc6dc1158e98c67c8b5bd97f14028942d15d542"} Jul 20 21:06:52.986973 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:52.986879 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-nft7r" podStartSLOduration=0.712182089 podStartE2EDuration="1.986861537s" totalImagesPullingTime="1.274679448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 21:06:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:06:52.986030284 +0000 UTC m=+252.311733227" watchObservedRunningTime="2026-07-20 21:06:52.986861537 +0000 UTC m=+252.312564479" Jul 20 21:06:53.012273 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:53.012236 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f09f6b9-6ed0-4165-bc40-706eb42f8f0d-metrics-certs\") pod \"network-metrics-daemon-t79t8\" (UID: \"0f09f6b9-6ed0-4165-bc40-706eb42f8f0d\") " pod="openshift-multus/network-metrics-daemon-t79t8" Jul 20 21:06:53.012438 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:53.012393 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 20 21:06:53.012499 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:53.012463 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f09f6b9-6ed0-4165-bc40-706eb42f8f0d-metrics-certs podName:0f09f6b9-6ed0-4165-bc40-706eb42f8f0d nodeName:}" failed. No retries permitted until 2026-07-20 21:08:55.012447659 +0000 UTC m=+374.338150598 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0f09f6b9-6ed0-4165-bc40-706eb42f8f0d-metrics-certs") pod "network-metrics-daemon-t79t8" (UID: "0f09f6b9-6ed0-4165-bc40-706eb42f8f0d") : secret "metrics-daemon-secret" not found Jul 20 21:06:53.112103 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:53.112061 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:06:53.227760 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:53.227718 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:06:53.945649 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:53.945564 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-lv5br" event={"ID":"a002a221-d9f5-42b1-b07b-4b6e946cf5c5","Type":"ContainerStarted","Data":"bbb35fdedc69040c786987b4b4dd9bf29df5ff9ad1693cccdd0f4dfd65302f04"} Jul 20 21:06:54.040498 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:54.040446 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-lv5br" podStartSLOduration=1.047135874 podStartE2EDuration="3.04043242s" totalImagesPullingTime="1.993296546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 21:06:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:06:54.039812669 +0000 UTC m=+253.365515612" watchObservedRunningTime="2026-07-20 21:06:54.04043242 +0000 UTC m=+253.366135399" Jul 20 21:06:54.729779 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:54.729731 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a82a3265-df2c-4d0f-8aaf-d2434825fc10-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-54chl\" (UID: \"a82a3265-df2c-4d0f-8aaf-d2434825fc10\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-54chl" Jul 20 21:06:54.729985 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:54.729886 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/da6da737-539a-4dbb-a8c1-994a4391da06-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-mxgbd\" (UID: \"da6da737-539a-4dbb-a8c1-994a4391da06\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-mxgbd" Jul 20 21:06:54.729985 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:54.729899 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 20 21:06:54.729985 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:54.729912 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8e63b84e-6cf9-467e-b114-9682be23c2b9-registry-tls\") pod \"image-registry-5748dd95bb-qhfpf\" (UID: \"8e63b84e-6cf9-467e-b114-9682be23c2b9\") " pod="openshift-image-registry/image-registry-5748dd95bb-qhfpf" Jul 20 21:06:54.730115 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:54.729991 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a82a3265-df2c-4d0f-8aaf-d2434825fc10-cluster-monitoring-operator-tls podName:a82a3265-df2c-4d0f-8aaf-d2434825fc10 nodeName:}" failed. No retries permitted until 2026-07-20 21:06:58.729975115 +0000 UTC m=+258.055678036 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a82a3265-df2c-4d0f-8aaf-d2434825fc10-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-54chl" (UID: "a82a3265-df2c-4d0f-8aaf-d2434825fc10") : secret "cluster-monitoring-operator-tls" not found Jul 20 21:06:54.730115 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:54.730011 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 20 21:06:54.730115 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:54.730073 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da6da737-539a-4dbb-a8c1-994a4391da06-samples-operator-tls podName:da6da737-539a-4dbb-a8c1-994a4391da06 nodeName:}" failed. No retries permitted until 2026-07-20 21:06:58.730060123 +0000 UTC m=+258.055763044 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/da6da737-539a-4dbb-a8c1-994a4391da06-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-mxgbd" (UID: "da6da737-539a-4dbb-a8c1-994a4391da06") : secret "samples-operator-tls" not found Jul 20 21:06:54.730115 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:54.730015 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 20 21:06:54.730115 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:54.730090 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5748dd95bb-qhfpf: secret "image-registry-tls" not found Jul 20 21:06:54.730115 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:54.730118 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e63b84e-6cf9-467e-b114-9682be23c2b9-registry-tls podName:8e63b84e-6cf9-467e-b114-9682be23c2b9 nodeName:}" failed. No retries permitted until 2026-07-20 21:06:58.73010955 +0000 UTC m=+258.055812475 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8e63b84e-6cf9-467e-b114-9682be23c2b9-registry-tls") pod "image-registry-5748dd95bb-qhfpf" (UID: "8e63b84e-6cf9-467e-b114-9682be23c2b9") : secret "image-registry-tls" not found Jul 20 21:06:55.193800 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:55.193765 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-59hpz"] Jul 20 21:06:55.195675 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:55.195658 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-59hpz" Jul 20 21:06:55.199039 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:55.199013 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 20 21:06:55.199893 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:55.199876 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 20 21:06:55.199893 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:55.199884 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-hh6fw\"" Jul 20 21:06:55.205591 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:55.205571 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:06:55.212833 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:55.212815 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-59hpz"] Jul 20 21:06:55.334757 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:55.334718 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcbp8\" (UniqueName: \"kubernetes.io/projected/b3db7a22-0821-4a68-bc79-1efbd85b5483-kube-api-access-bcbp8\") pod \"migrator-7cc9c7ffd7-59hpz\" (UID: \"b3db7a22-0821-4a68-bc79-1efbd85b5483\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-59hpz" Jul 20 21:06:55.340868 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:55.340829 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:06:55.435916 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:55.435885 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcbp8\" (UniqueName: \"kubernetes.io/projected/b3db7a22-0821-4a68-bc79-1efbd85b5483-kube-api-access-bcbp8\") pod \"migrator-7cc9c7ffd7-59hpz\" (UID: \"b3db7a22-0821-4a68-bc79-1efbd85b5483\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-59hpz" Jul 20 21:06:55.447931 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:55.447860 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcbp8\" (UniqueName: \"kubernetes.io/projected/b3db7a22-0821-4a68-bc79-1efbd85b5483-kube-api-access-bcbp8\") pod \"migrator-7cc9c7ffd7-59hpz\" (UID: \"b3db7a22-0821-4a68-bc79-1efbd85b5483\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-59hpz" Jul 20 21:06:55.503894 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:55.503865 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-59hpz" Jul 20 21:06:55.614039 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:55.614012 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rq8g2"] Jul 20 21:06:55.616215 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:55.616197 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rq8g2" Jul 20 21:06:55.620486 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:55.620451 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-136-32.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-136-32.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Jul 20 21:06:55.620637 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:55.620596 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"insights-runtime-extractor-rq8g2\" is forbidden: User \"system:node:ip-10-0-136-32.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-136-32.ec2.internal' and this object" podUID="399a1523-41ed-4080-9321-66533820eded" pod="openshift-insights/insights-runtime-extractor-rq8g2" Jul 20 21:06:55.620897 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:55.620868 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"insights-runtime-extractor-tls\" is forbidden: User \"system:node:ip-10-0-136-32.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-136-32.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" type="*v1.Secret" Jul 20 21:06:55.621002 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:55.620951 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-rbac-proxy\" is forbidden: User \"system:node:ip-10-0-136-32.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-136-32.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" type="*v1.ConfigMap" Jul 20 21:06:55.621298 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:55.621278 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"insights-runtime-extractor-sa-dockercfg-kgm4r\" is forbidden: User \"system:node:ip-10-0-136-32.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-136-32.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kgm4r\"" type="*v1.Secret" Jul 20 21:06:55.621443 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:55.621427 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-136-32.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-136-32.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Jul 20 21:06:55.650030 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:06:55.649996 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3db7a22_0821_4a68_bc79_1efbd85b5483.slice/crio-1d02f5526e866c691813c9b98a2f7990e215982d00759e0288c7e169870b87a2 WatchSource:0}: Error finding container 1d02f5526e866c691813c9b98a2f7990e215982d00759e0288c7e169870b87a2: Status 404 returned error can't find the container with id 1d02f5526e866c691813c9b98a2f7990e215982d00759e0288c7e169870b87a2 Jul 20 21:06:55.650175 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:55.650038 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-59hpz"] Jul 20 21:06:55.692518 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:55.692479 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rq8g2"] Jul 20 21:06:55.737671 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:55.737599 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/399a1523-41ed-4080-9321-66533820eded-data-volume\") pod \"insights-runtime-extractor-rq8g2\" (UID: \"399a1523-41ed-4080-9321-66533820eded\") " pod="openshift-insights/insights-runtime-extractor-rq8g2" Jul 20 21:06:55.737794 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:55.737748 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/399a1523-41ed-4080-9321-66533820eded-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rq8g2\" (UID: \"399a1523-41ed-4080-9321-66533820eded\") " pod="openshift-insights/insights-runtime-extractor-rq8g2" Jul 20 21:06:55.737794 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:55.737779 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv2g6\" (UniqueName: \"kubernetes.io/projected/399a1523-41ed-4080-9321-66533820eded-kube-api-access-pv2g6\") pod \"insights-runtime-extractor-rq8g2\" (UID: \"399a1523-41ed-4080-9321-66533820eded\") " pod="openshift-insights/insights-runtime-extractor-rq8g2" Jul 20 21:06:55.737886 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:55.737801 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/399a1523-41ed-4080-9321-66533820eded-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rq8g2\" (UID: \"399a1523-41ed-4080-9321-66533820eded\") " pod="openshift-insights/insights-runtime-extractor-rq8g2" Jul 20 21:06:55.737941 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:55.737878 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/399a1523-41ed-4080-9321-66533820eded-crio-socket\") pod \"insights-runtime-extractor-rq8g2\" (UID: \"399a1523-41ed-4080-9321-66533820eded\") " pod="openshift-insights/insights-runtime-extractor-rq8g2" Jul 20 21:06:55.753858 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:55.753832 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:06:55.838358 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:55.838326 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/399a1523-41ed-4080-9321-66533820eded-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rq8g2\" (UID: \"399a1523-41ed-4080-9321-66533820eded\") " pod="openshift-insights/insights-runtime-extractor-rq8g2" Jul 20 21:06:55.838358 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:55.838356 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv2g6\" (UniqueName: \"kubernetes.io/projected/399a1523-41ed-4080-9321-66533820eded-kube-api-access-pv2g6\") pod \"insights-runtime-extractor-rq8g2\" (UID: \"399a1523-41ed-4080-9321-66533820eded\") " pod="openshift-insights/insights-runtime-extractor-rq8g2" Jul 20 21:06:55.838534 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:55.838379 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/399a1523-41ed-4080-9321-66533820eded-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rq8g2\" (UID: \"399a1523-41ed-4080-9321-66533820eded\") " pod="openshift-insights/insights-runtime-extractor-rq8g2" Jul 20 21:06:55.838534 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:55.838398 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/399a1523-41ed-4080-9321-66533820eded-crio-socket\") pod \"insights-runtime-extractor-rq8g2\" (UID: \"399a1523-41ed-4080-9321-66533820eded\") " pod="openshift-insights/insights-runtime-extractor-rq8g2" Jul 20 21:06:55.838534 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:55.838436 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/399a1523-41ed-4080-9321-66533820eded-data-volume\") pod \"insights-runtime-extractor-rq8g2\" (UID: \"399a1523-41ed-4080-9321-66533820eded\") " pod="openshift-insights/insights-runtime-extractor-rq8g2" Jul 20 21:06:55.838534 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:55.838498 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/399a1523-41ed-4080-9321-66533820eded-crio-socket\") pod \"insights-runtime-extractor-rq8g2\" (UID: \"399a1523-41ed-4080-9321-66533820eded\") " pod="openshift-insights/insights-runtime-extractor-rq8g2" Jul 20 21:06:55.838725 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:55.838712 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/399a1523-41ed-4080-9321-66533820eded-data-volume\") pod \"insights-runtime-extractor-rq8g2\" (UID: \"399a1523-41ed-4080-9321-66533820eded\") " pod="openshift-insights/insights-runtime-extractor-rq8g2" Jul 20 21:06:55.913413 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:55.913383 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:06:55.951682 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:55.951651 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-59hpz" event={"ID":"b3db7a22-0821-4a68-bc79-1efbd85b5483","Type":"ContainerStarted","Data":"1d02f5526e866c691813c9b98a2f7990e215982d00759e0288c7e169870b87a2"} Jul 20 21:06:56.005237 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:56.005167 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:06:56.106085 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:56.106051 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:06:56.477130 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:56.477098 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 20 21:06:56.479222 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:56.479203 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 21:06:56.479331 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:56.479274 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/399a1523-41ed-4080-9321-66533820eded-insights-runtime-extractor-tls podName:399a1523-41ed-4080-9321-66533820eded nodeName:}" failed. No retries permitted until 2026-07-20 21:06:56.979253408 +0000 UTC m=+256.304956335 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/399a1523-41ed-4080-9321-66533820eded-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rq8g2" (UID: "399a1523-41ed-4080-9321-66533820eded") : secret "insights-runtime-extractor-tls" not found Jul 20 21:06:56.618295 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:56.618267 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 20 21:06:56.719438 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:56.719385 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 20 21:06:56.728797 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:56.728716 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 20 21:06:56.728959 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:56.728815 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/399a1523-41ed-4080-9321-66533820eded-kube-rbac-proxy-cm podName:399a1523-41ed-4080-9321-66533820eded nodeName:}" failed. No retries permitted until 2026-07-20 21:06:57.228791989 +0000 UTC m=+256.554494947 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/399a1523-41ed-4080-9321-66533820eded-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rq8g2" (UID: "399a1523-41ed-4080-9321-66533820eded") : configmap "kube-rbac-proxy" not found Jul 20 21:06:56.744437 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:56.744405 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:06:56.812479 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:56.812457 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 20 21:06:56.820710 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:56.820681 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv2g6\" (UniqueName: \"kubernetes.io/projected/399a1523-41ed-4080-9321-66533820eded-kube-api-access-pv2g6\") pod \"insights-runtime-extractor-rq8g2\" (UID: \"399a1523-41ed-4080-9321-66533820eded\") " pod="openshift-insights/insights-runtime-extractor-rq8g2" Jul 20 21:06:56.837571 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:56.837554 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kgm4r\"" Jul 20 21:06:56.840394 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:56.840368 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:06:56.956576 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:56.956545 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-59hpz" event={"ID":"b3db7a22-0821-4a68-bc79-1efbd85b5483","Type":"ContainerStarted","Data":"c1ba5f46a31830cfc0e9dca4a84c25bd7a67fbe23d516064a368804ae533516c"} Jul 20 21:06:56.956788 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:56.956614 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-59hpz" event={"ID":"b3db7a22-0821-4a68-bc79-1efbd85b5483","Type":"ContainerStarted","Data":"6cd3e4b8c45eccdc08382cac17e5b5dd438118ccbb1f43053497e7e5939ef852"} Jul 20 21:06:56.979552 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:56.979352 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-59hpz" podStartSLOduration=0.855618937 podStartE2EDuration="1.979335781s" totalImagesPullingTime="1.123716844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 21:06:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:06:56.978806643 +0000 UTC m=+256.304509586" watchObservedRunningTime="2026-07-20 21:06:56.979335781 +0000 UTC m=+256.305038723" Jul 20 21:06:57.047749 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:57.047714 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/399a1523-41ed-4080-9321-66533820eded-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rq8g2\" (UID: \"399a1523-41ed-4080-9321-66533820eded\") " pod="openshift-insights/insights-runtime-extractor-rq8g2" Jul 20 21:06:57.047910 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:57.047879 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 21:06:57.047970 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:57.047938 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/399a1523-41ed-4080-9321-66533820eded-insights-runtime-extractor-tls podName:399a1523-41ed-4080-9321-66533820eded nodeName:}" failed. No retries permitted until 2026-07-20 21:06:58.047922569 +0000 UTC m=+257.373625490 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/399a1523-41ed-4080-9321-66533820eded-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rq8g2" (UID: "399a1523-41ed-4080-9321-66533820eded") : secret "insights-runtime-extractor-tls" not found Jul 20 21:06:57.092866 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:57.092836 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:06:57.166226 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:57.166198 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:06:57.250184 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:57.250113 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/399a1523-41ed-4080-9321-66533820eded-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rq8g2\" (UID: \"399a1523-41ed-4080-9321-66533820eded\") " pod="openshift-insights/insights-runtime-extractor-rq8g2" Jul 20 21:06:57.250299 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:57.250203 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 20 21:06:57.250299 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:57.250252 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/399a1523-41ed-4080-9321-66533820eded-kube-rbac-proxy-cm podName:399a1523-41ed-4080-9321-66533820eded nodeName:}" failed. No retries permitted until 2026-07-20 21:06:58.250240419 +0000 UTC m=+257.575943339 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/399a1523-41ed-4080-9321-66533820eded-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rq8g2" (UID: "399a1523-41ed-4080-9321-66533820eded") : configmap "kube-rbac-proxy" not found Jul 20 21:06:58.056345 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:58.056309 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/399a1523-41ed-4080-9321-66533820eded-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rq8g2\" (UID: \"399a1523-41ed-4080-9321-66533820eded\") " pod="openshift-insights/insights-runtime-extractor-rq8g2" Jul 20 21:06:58.057656 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:58.056841 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 21:06:58.057656 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:58.056952 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/399a1523-41ed-4080-9321-66533820eded-insights-runtime-extractor-tls podName:399a1523-41ed-4080-9321-66533820eded nodeName:}" failed. No retries permitted until 2026-07-20 21:07:00.056931411 +0000 UTC m=+259.382634334 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/399a1523-41ed-4080-9321-66533820eded-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rq8g2" (UID: "399a1523-41ed-4080-9321-66533820eded") : secret "insights-runtime-extractor-tls" not found Jul 20 21:06:58.262619 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:58.262579 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/399a1523-41ed-4080-9321-66533820eded-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rq8g2\" (UID: \"399a1523-41ed-4080-9321-66533820eded\") " pod="openshift-insights/insights-runtime-extractor-rq8g2" Jul 20 21:06:58.262784 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:58.262733 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 20 21:06:58.262827 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:58.262796 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/399a1523-41ed-4080-9321-66533820eded-kube-rbac-proxy-cm podName:399a1523-41ed-4080-9321-66533820eded nodeName:}" failed. No retries permitted until 2026-07-20 21:07:00.262781707 +0000 UTC m=+259.588484627 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/399a1523-41ed-4080-9321-66533820eded-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rq8g2" (UID: "399a1523-41ed-4080-9321-66533820eded") : configmap "kube-rbac-proxy" not found Jul 20 21:06:58.351189 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:58.351098 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:06:58.430648 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:58.430624 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qcsfr_dde8ddcb-7c86-4c9d-bd00-11129c8ca35e/dns-node-resolver/0.log" Jul 20 21:06:58.437548 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:58.437525 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:06:58.765553 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:58.765526 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8e63b84e-6cf9-467e-b114-9682be23c2b9-registry-tls\") pod \"image-registry-5748dd95bb-qhfpf\" (UID: \"8e63b84e-6cf9-467e-b114-9682be23c2b9\") " pod="openshift-image-registry/image-registry-5748dd95bb-qhfpf" Jul 20 21:06:58.765677 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:58.765584 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/da6da737-539a-4dbb-a8c1-994a4391da06-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-mxgbd\" (UID: \"da6da737-539a-4dbb-a8c1-994a4391da06\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-mxgbd" Jul 20 21:06:58.765677 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:58.765617 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a82a3265-df2c-4d0f-8aaf-d2434825fc10-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-54chl\" (UID: \"a82a3265-df2c-4d0f-8aaf-d2434825fc10\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-54chl" Jul 20 21:06:58.765762 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:58.765686 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 20 21:06:58.765762 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:58.765703 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5748dd95bb-qhfpf: secret "image-registry-tls" not found Jul 20 21:06:58.765762 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:58.765725 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 20 21:06:58.765762 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:58.765728 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 20 21:06:58.765762 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:58.765756 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e63b84e-6cf9-467e-b114-9682be23c2b9-registry-tls podName:8e63b84e-6cf9-467e-b114-9682be23c2b9 nodeName:}" failed. No retries permitted until 2026-07-20 21:07:06.76574084 +0000 UTC m=+266.091443783 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8e63b84e-6cf9-467e-b114-9682be23c2b9-registry-tls") pod "image-registry-5748dd95bb-qhfpf" (UID: "8e63b84e-6cf9-467e-b114-9682be23c2b9") : secret "image-registry-tls" not found Jul 20 21:06:58.765911 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:58.765774 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da6da737-539a-4dbb-a8c1-994a4391da06-samples-operator-tls podName:da6da737-539a-4dbb-a8c1-994a4391da06 nodeName:}" failed. No retries permitted until 2026-07-20 21:07:06.765762464 +0000 UTC m=+266.091465388 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/da6da737-539a-4dbb-a8c1-994a4391da06-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-mxgbd" (UID: "da6da737-539a-4dbb-a8c1-994a4391da06") : secret "samples-operator-tls" not found Jul 20 21:06:58.765911 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:06:58.765788 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a82a3265-df2c-4d0f-8aaf-d2434825fc10-cluster-monitoring-operator-tls podName:a82a3265-df2c-4d0f-8aaf-d2434825fc10 nodeName:}" failed. No retries permitted until 2026-07-20 21:07:06.765781895 +0000 UTC m=+266.091484815 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a82a3265-df2c-4d0f-8aaf-d2434825fc10-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-54chl" (UID: "a82a3265-df2c-4d0f-8aaf-d2434825fc10") : secret "cluster-monitoring-operator-tls" not found Jul 20 21:06:59.630873 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:59.630841 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:06:59.637734 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:59.637716 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lgq8n_ef68231f-0506-4450-9fcb-bc14af3b7d76/node-ca/0.log" Jul 20 21:06:59.767150 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:59.767104 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:06:59.892953 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:06:59.892880 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:00.074024 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:00.073991 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:00.076081 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:00.076052 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/399a1523-41ed-4080-9321-66533820eded-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rq8g2\" (UID: \"399a1523-41ed-4080-9321-66533820eded\") " pod="openshift-insights/insights-runtime-extractor-rq8g2" Jul 20 21:07:00.076223 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:07:00.076208 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 21:07:00.076276 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:07:00.076267 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/399a1523-41ed-4080-9321-66533820eded-insights-runtime-extractor-tls podName:399a1523-41ed-4080-9321-66533820eded nodeName:}" failed. No retries permitted until 2026-07-20 21:07:04.076252155 +0000 UTC m=+263.401955076 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/399a1523-41ed-4080-9321-66533820eded-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rq8g2" (UID: "399a1523-41ed-4080-9321-66533820eded") : secret "insights-runtime-extractor-tls" not found Jul 20 21:07:00.255694 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:00.255612 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:00.278050 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:00.278018 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/399a1523-41ed-4080-9321-66533820eded-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rq8g2\" (UID: \"399a1523-41ed-4080-9321-66533820eded\") " pod="openshift-insights/insights-runtime-extractor-rq8g2" Jul 20 21:07:00.278215 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:07:00.278113 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 20 21:07:00.278215 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:07:00.278199 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/399a1523-41ed-4080-9321-66533820eded-kube-rbac-proxy-cm podName:399a1523-41ed-4080-9321-66533820eded nodeName:}" failed. No retries permitted until 2026-07-20 21:07:04.278181055 +0000 UTC m=+263.603883979 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/399a1523-41ed-4080-9321-66533820eded-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rq8g2" (UID: "399a1523-41ed-4080-9321-66533820eded") : configmap "kube-rbac-proxy" not found Jul 20 21:07:00.437901 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:00.437874 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jd2kv_d1bb6ff3-c7d2-4be7-9e62-983c4f6bbff4/serve-healthcheck-canary/0.log" Jul 20 21:07:00.472097 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:00.472066 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:00.649274 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:00.649246 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-59hpz_b3db7a22-0821-4a68-bc79-1efbd85b5483/migrator/0.log" Jul 20 21:07:00.841118 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:00.841089 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-59hpz_b3db7a22-0821-4a68-bc79-1efbd85b5483/graceful-termination/0.log" Jul 20 21:07:01.031257 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:01.031234 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-lv5br_a002a221-d9f5-42b1-b07b-4b6e946cf5c5/kube-storage-version-migrator-operator/0.log" Jul 20 21:07:01.593862 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:01.593829 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:01.832298 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:01.832265 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:01.985186 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:01.985089 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:02.191227 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:02.191197 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:02.407748 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:02.407720 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:03.604731 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:03.604699 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:03.709339 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:03.709305 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:03.818390 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:03.818362 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:03.999080 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:03.998989 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:04.111799 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:04.111768 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/399a1523-41ed-4080-9321-66533820eded-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rq8g2\" (UID: \"399a1523-41ed-4080-9321-66533820eded\") " pod="openshift-insights/insights-runtime-extractor-rq8g2" Jul 20 21:07:04.111955 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:07:04.111905 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 21:07:04.111997 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:07:04.111967 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/399a1523-41ed-4080-9321-66533820eded-insights-runtime-extractor-tls podName:399a1523-41ed-4080-9321-66533820eded nodeName:}" failed. No retries permitted until 2026-07-20 21:07:12.111951085 +0000 UTC m=+271.437654007 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/399a1523-41ed-4080-9321-66533820eded-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rq8g2" (UID: "399a1523-41ed-4080-9321-66533820eded") : secret "insights-runtime-extractor-tls" not found Jul 20 21:07:04.190613 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:04.190582 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:04.286424 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:04.286395 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:04.313798 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:04.313770 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/399a1523-41ed-4080-9321-66533820eded-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rq8g2\" (UID: \"399a1523-41ed-4080-9321-66533820eded\") " pod="openshift-insights/insights-runtime-extractor-rq8g2" Jul 20 21:07:04.313920 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:07:04.313882 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 20 21:07:04.313964 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:07:04.313936 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/399a1523-41ed-4080-9321-66533820eded-kube-rbac-proxy-cm podName:399a1523-41ed-4080-9321-66533820eded nodeName:}" failed. No retries permitted until 2026-07-20 21:07:12.313921683 +0000 UTC m=+271.639624606 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/399a1523-41ed-4080-9321-66533820eded-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rq8g2" (UID: "399a1523-41ed-4080-9321-66533820eded") : configmap "kube-rbac-proxy" not found Jul 20 21:07:04.411281 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:04.411252 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:05.490611 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:05.490575 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:05.817335 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:05.817307 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:05.977914 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:05.977882 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:06.232319 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:06.232229 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:06.362325 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:06.362299 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:06.465273 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:06.465241 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:06.836904 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:06.836858 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/da6da737-539a-4dbb-a8c1-994a4391da06-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-mxgbd\" (UID: \"da6da737-539a-4dbb-a8c1-994a4391da06\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-mxgbd" Jul 20 21:07:06.837406 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:06.836920 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a82a3265-df2c-4d0f-8aaf-d2434825fc10-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-54chl\" (UID: \"a82a3265-df2c-4d0f-8aaf-d2434825fc10\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-54chl" Jul 20 21:07:06.837406 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:06.836982 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8e63b84e-6cf9-467e-b114-9682be23c2b9-registry-tls\") pod \"image-registry-5748dd95bb-qhfpf\" (UID: \"8e63b84e-6cf9-467e-b114-9682be23c2b9\") " pod="openshift-image-registry/image-registry-5748dd95bb-qhfpf" Jul 20 21:07:06.839308 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:06.839281 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/da6da737-539a-4dbb-a8c1-994a4391da06-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-mxgbd\" (UID: \"da6da737-539a-4dbb-a8c1-994a4391da06\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-mxgbd" Jul 20 21:07:06.839455 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:06.839312 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a82a3265-df2c-4d0f-8aaf-d2434825fc10-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-54chl\" (UID: \"a82a3265-df2c-4d0f-8aaf-d2434825fc10\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-54chl" Jul 20 21:07:06.839455 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:06.839413 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8e63b84e-6cf9-467e-b114-9682be23c2b9-registry-tls\") pod \"image-registry-5748dd95bb-qhfpf\" (UID: \"8e63b84e-6cf9-467e-b114-9682be23c2b9\") " pod="openshift-image-registry/image-registry-5748dd95bb-qhfpf" Jul 20 21:07:06.906412 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:06.906384 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-54chl" Jul 20 21:07:06.910922 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:06.910908 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-mxgbd" Jul 20 21:07:06.944724 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:06.944658 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5748dd95bb-qhfpf" Jul 20 21:07:07.049898 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:07.049879 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-54chl"] Jul 20 21:07:07.052969 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:07:07.052943 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda82a3265_df2c_4d0f_8aaf_d2434825fc10.slice/crio-08af3f25207a85b07ba17433449b4092c7f8c9c1b0435749f54e4f966694f3fa WatchSource:0}: Error finding container 08af3f25207a85b07ba17433449b4092c7f8c9c1b0435749f54e4f966694f3fa: Status 404 returned error can't find the container with id 08af3f25207a85b07ba17433449b4092c7f8c9c1b0435749f54e4f966694f3fa Jul 20 21:07:07.067127 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:07.067101 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-mxgbd"] Jul 20 21:07:07.096153 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:07.096112 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5748dd95bb-qhfpf"] Jul 20 21:07:07.099036 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:07:07.099016 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e63b84e_6cf9_467e_b114_9682be23c2b9.slice/crio-79c23fa2896e9f2766d62d028bb2e47435dfcd4c92c2fefb679017c23e8a7467 WatchSource:0}: Error finding container 79c23fa2896e9f2766d62d028bb2e47435dfcd4c92c2fefb679017c23e8a7467: Status 404 returned error can't find the container with id 79c23fa2896e9f2766d62d028bb2e47435dfcd4c92c2fefb679017c23e8a7467 Jul 20 21:07:07.368118 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:07.368045 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:07.601770 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:07.601738 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:07.820771 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:07.820739 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:07.991622 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:07.991542 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5748dd95bb-qhfpf" event={"ID":"8e63b84e-6cf9-467e-b114-9682be23c2b9","Type":"ContainerStarted","Data":"6d706271b8fa258ec214c6bf40bb0a4a51a81f1cbcce92f7195a429afc34cb00"} Jul 20 21:07:07.991622 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:07.991605 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5748dd95bb-qhfpf" event={"ID":"8e63b84e-6cf9-467e-b114-9682be23c2b9","Type":"ContainerStarted","Data":"79c23fa2896e9f2766d62d028bb2e47435dfcd4c92c2fefb679017c23e8a7467"} Jul 20 21:07:07.992168 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:07.991633 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5748dd95bb-qhfpf" Jul 20 21:07:07.993151 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:07.993108 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-mxgbd" event={"ID":"da6da737-539a-4dbb-a8c1-994a4391da06","Type":"ContainerStarted","Data":"3b5db0142f63ff95446762f4936a4e8008fc3bf359a316e74d88ad4366c44a36"} Jul 20 21:07:07.994526 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:07.994498 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-54chl" event={"ID":"a82a3265-df2c-4d0f-8aaf-d2434825fc10","Type":"ContainerStarted","Data":"08af3f25207a85b07ba17433449b4092c7f8c9c1b0435749f54e4f966694f3fa"} Jul 20 21:07:08.013589 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:08.013505 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5748dd95bb-qhfpf" podStartSLOduration=18.013486039 podStartE2EDuration="18.013486039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 21:06:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:07:08.013232546 +0000 UTC m=+267.338935491" watchObservedRunningTime="2026-07-20 21:07:08.013486039 +0000 UTC m=+267.339188961" Jul 20 21:07:08.162523 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:08.162444 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:08.272675 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:08.272639 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:08.578007 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:08.577977 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:09.255049 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:09.255026 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:09.696316 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:09.696281 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:09.818482 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:09.818452 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:09.929354 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:09.929319 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:10.006604 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:10.006505 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-mxgbd" event={"ID":"da6da737-539a-4dbb-a8c1-994a4391da06","Type":"ContainerStarted","Data":"aff50f73199de9aaeef02afe4e9ef30a251f374df66f6cbf57fdf9a2e164ec9d"} Jul 20 21:07:10.006604 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:10.006600 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-mxgbd" event={"ID":"da6da737-539a-4dbb-a8c1-994a4391da06","Type":"ContainerStarted","Data":"6f2822d7e183f6acc6dfdeacaf0fe32e90ee8445217926d526b55bfd61565710"} Jul 20 21:07:10.007937 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:10.007911 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-54chl" event={"ID":"a82a3265-df2c-4d0f-8aaf-d2434825fc10","Type":"ContainerStarted","Data":"3b91f4866930fe5414b9233a5d6aecf36e4befdaf82e5cd9413a3b6f60526c45"} Jul 20 21:07:10.026769 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:10.026737 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:10.065925 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:10.065876 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-mxgbd" podStartSLOduration=17.929671375 podStartE2EDuration="20.065861132s" totalImagesPullingTime="2.136189757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 21:06:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:07:10.064686752 +0000 UTC m=+269.390389696" watchObservedRunningTime="2026-07-20 21:07:10.065861132 +0000 UTC m=+269.391564075" Jul 20 21:07:10.105040 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:10.104987 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-54chl" podStartSLOduration=17.919859426 podStartE2EDuration="20.104966677s" totalImagesPullingTime="2.185107251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 21:06:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:07:10.102708242 +0000 UTC m=+269.428411187" watchObservedRunningTime="2026-07-20 21:07:10.104966677 +0000 UTC m=+269.430669622" Jul 20 21:07:10.186528 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:10.186494 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:10.380356 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:10.380321 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:10.739841 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:10.739754 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:10.805622 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:10.805588 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:11.181236 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:11.181207 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:12.179965 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:12.179926 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/399a1523-41ed-4080-9321-66533820eded-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rq8g2\" (UID: \"399a1523-41ed-4080-9321-66533820eded\") " pod="openshift-insights/insights-runtime-extractor-rq8g2" Jul 20 21:07:12.182247 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:12.182226 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/399a1523-41ed-4080-9321-66533820eded-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rq8g2\" (UID: \"399a1523-41ed-4080-9321-66533820eded\") " pod="openshift-insights/insights-runtime-extractor-rq8g2" Jul 20 21:07:12.381661 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:12.381619 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/399a1523-41ed-4080-9321-66533820eded-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rq8g2\" (UID: \"399a1523-41ed-4080-9321-66533820eded\") " pod="openshift-insights/insights-runtime-extractor-rq8g2" Jul 20 21:07:12.382181 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:12.382160 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/399a1523-41ed-4080-9321-66533820eded-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rq8g2\" (UID: \"399a1523-41ed-4080-9321-66533820eded\") " pod="openshift-insights/insights-runtime-extractor-rq8g2" Jul 20 21:07:12.424850 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:12.424826 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rq8g2" Jul 20 21:07:12.556443 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:12.556411 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rq8g2"] Jul 20 21:07:12.559331 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:07:12.559296 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod399a1523_41ed_4080_9321_66533820eded.slice/crio-826a4b8849d9016c0ef0457509902375bbbf3dc06d879587e1989b6ca33eda31 WatchSource:0}: Error finding container 826a4b8849d9016c0ef0457509902375bbbf3dc06d879587e1989b6ca33eda31: Status 404 returned error can't find the container with id 826a4b8849d9016c0ef0457509902375bbbf3dc06d879587e1989b6ca33eda31 Jul 20 21:07:13.022173 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:13.022121 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rq8g2" event={"ID":"399a1523-41ed-4080-9321-66533820eded","Type":"ContainerStarted","Data":"826a4b8849d9016c0ef0457509902375bbbf3dc06d879587e1989b6ca33eda31"} Jul 20 21:07:13.821832 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:13.821800 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:13.901331 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:13.901260 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:14.044491 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:14.044456 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:14.221722 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:14.221627 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:14.288754 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:14.288721 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:14.623113 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:14.623086 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:14.741330 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:14.741294 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:15.456858 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:15.456816 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:15.650484 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:15.650448 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:15.750939 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:15.750844 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:15.850986 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:15.850952 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:18.041922 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:18.041885 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rq8g2" event={"ID":"399a1523-41ed-4080-9321-66533820eded","Type":"ContainerStarted","Data":"e27619c48c1c069c59b45845b30bd4763cbfb49a21bfe2c969a6f3ec781c8f45"} Jul 20 21:07:20.064761 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:20.064733 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-fphgz"] Jul 20 21:07:20.069795 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:20.069774 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-fphgz" Jul 20 21:07:20.073972 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:20.073922 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-rfrvr\"" Jul 20 21:07:20.073972 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:20.073947 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 20 21:07:20.083876 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:20.083845 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-fphgz"] Jul 20 21:07:20.096630 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:20.096607 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5748dd95bb-qhfpf"] Jul 20 21:07:20.131190 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:20.131160 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b8372209-79c7-49f4-89eb-c9ad3846e1ae-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-fphgz\" (UID: \"b8372209-79c7-49f4-89eb-c9ad3846e1ae\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-fphgz" Jul 20 21:07:20.209686 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:20.209606 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:20.232366 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:20.232328 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b8372209-79c7-49f4-89eb-c9ad3846e1ae-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-fphgz\" (UID: \"b8372209-79c7-49f4-89eb-c9ad3846e1ae\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-fphgz" Jul 20 21:07:20.234946 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:20.234917 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b8372209-79c7-49f4-89eb-c9ad3846e1ae-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-fphgz\" (UID: \"b8372209-79c7-49f4-89eb-c9ad3846e1ae\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-fphgz" Jul 20 21:07:20.311958 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:20.311926 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:20.381825 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:20.381748 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-fphgz" Jul 20 21:07:20.754778 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:20.754751 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-fphgz"] Jul 20 21:07:20.757320 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:07:20.757294 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8372209_79c7_49f4_89eb_c9ad3846e1ae.slice/crio-57a9ce40545efcf55ec3decc354ddd75d3457f7812bbce0a8e0e15727e50216a WatchSource:0}: Error finding container 57a9ce40545efcf55ec3decc354ddd75d3457f7812bbce0a8e0e15727e50216a: Status 404 returned error can't find the container with id 57a9ce40545efcf55ec3decc354ddd75d3457f7812bbce0a8e0e15727e50216a Jul 20 21:07:21.052419 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:21.052390 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-fphgz" event={"ID":"b8372209-79c7-49f4-89eb-c9ad3846e1ae","Type":"ContainerStarted","Data":"57a9ce40545efcf55ec3decc354ddd75d3457f7812bbce0a8e0e15727e50216a"} Jul 20 21:07:21.054015 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:21.053994 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rq8g2" event={"ID":"399a1523-41ed-4080-9321-66533820eded","Type":"ContainerStarted","Data":"ddf7522da58da480a588f4b1cb5098b5b72e83d2b36f3b5e1052a041bdf1ec9d"} Jul 20 21:07:22.727024 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:07:22.726976 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-zp575" podUID="7cc5fdb7-149c-47f7-805a-cadc7c5f2599" Jul 20 21:07:23.062782 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:23.062730 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-fphgz" event={"ID":"b8372209-79c7-49f4-89eb-c9ad3846e1ae","Type":"ContainerStarted","Data":"fcd119074a179e65e4c9bedee591dd6b8f5ac0cf50875df931cc4d329829f8e7"} Jul 20 21:07:23.063071 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:23.062786 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zp575" Jul 20 21:07:23.063071 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:23.063047 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-fphgz" Jul 20 21:07:23.068187 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:23.068165 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-fphgz" Jul 20 21:07:23.081777 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:23.081724 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-fphgz" podStartSLOduration=1.7407962270000001 podStartE2EDuration="3.081708294s" totalImagesPullingTime="1.340912067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 21:07:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:07:23.079385119 +0000 UTC m=+282.405088065" watchObservedRunningTime="2026-07-20 21:07:23.081708294 +0000 UTC m=+282.407411237" Jul 20 21:07:23.877310 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:23.877273 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-75c69c9dc9-vl49s"] Jul 20 21:07:23.897611 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:23.897578 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75c69c9dc9-vl49s"] Jul 20 21:07:23.897771 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:23.897702 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75c69c9dc9-vl49s" Jul 20 21:07:23.900448 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:23.900421 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 20 21:07:23.900448 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:23.900432 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 20 21:07:23.900650 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:23.900431 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 20 21:07:23.901583 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:23.901424 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 20 21:07:23.901583 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:23.901454 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 20 21:07:23.901583 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:23.901461 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 20 21:07:23.901583 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:23.901466 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 20 21:07:23.901583 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:23.901526 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-98t82\"" Jul 20 21:07:23.949841 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:23.949801 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:23.960888 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:23.960800 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/280d1cb0-534d-49b9-b267-7882dc079daa-console-serving-cert\") pod \"console-75c69c9dc9-vl49s\" (UID: \"280d1cb0-534d-49b9-b267-7882dc079daa\") " pod="openshift-console/console-75c69c9dc9-vl49s" Jul 20 21:07:23.960888 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:23.960841 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/280d1cb0-534d-49b9-b267-7882dc079daa-service-ca\") pod \"console-75c69c9dc9-vl49s\" (UID: \"280d1cb0-534d-49b9-b267-7882dc079daa\") " pod="openshift-console/console-75c69c9dc9-vl49s" Jul 20 21:07:23.960888 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:23.960865 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g94n\" (UniqueName: \"kubernetes.io/projected/280d1cb0-534d-49b9-b267-7882dc079daa-kube-api-access-6g94n\") pod \"console-75c69c9dc9-vl49s\" (UID: \"280d1cb0-534d-49b9-b267-7882dc079daa\") " pod="openshift-console/console-75c69c9dc9-vl49s" Jul 20 21:07:23.961219 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:23.960976 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/280d1cb0-534d-49b9-b267-7882dc079daa-console-config\") pod \"console-75c69c9dc9-vl49s\" (UID: \"280d1cb0-534d-49b9-b267-7882dc079daa\") " pod="openshift-console/console-75c69c9dc9-vl49s" Jul 20 21:07:23.961219 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:23.961012 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/280d1cb0-534d-49b9-b267-7882dc079daa-oauth-serving-cert\") pod \"console-75c69c9dc9-vl49s\" (UID: \"280d1cb0-534d-49b9-b267-7882dc079daa\") " pod="openshift-console/console-75c69c9dc9-vl49s" Jul 20 21:07:23.961328 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:23.961246 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/280d1cb0-534d-49b9-b267-7882dc079daa-console-oauth-config\") pod \"console-75c69c9dc9-vl49s\" (UID: \"280d1cb0-534d-49b9-b267-7882dc079daa\") " pod="openshift-console/console-75c69c9dc9-vl49s" Jul 20 21:07:24.062446 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:24.062409 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/280d1cb0-534d-49b9-b267-7882dc079daa-console-serving-cert\") pod \"console-75c69c9dc9-vl49s\" (UID: \"280d1cb0-534d-49b9-b267-7882dc079daa\") " pod="openshift-console/console-75c69c9dc9-vl49s" Jul 20 21:07:24.062446 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:24.062451 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/280d1cb0-534d-49b9-b267-7882dc079daa-service-ca\") pod \"console-75c69c9dc9-vl49s\" (UID: \"280d1cb0-534d-49b9-b267-7882dc079daa\") " pod="openshift-console/console-75c69c9dc9-vl49s" Jul 20 21:07:24.062693 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:24.062470 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g94n\" (UniqueName: \"kubernetes.io/projected/280d1cb0-534d-49b9-b267-7882dc079daa-kube-api-access-6g94n\") pod \"console-75c69c9dc9-vl49s\" (UID: \"280d1cb0-534d-49b9-b267-7882dc079daa\") " pod="openshift-console/console-75c69c9dc9-vl49s" Jul 20 21:07:24.062693 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:24.062568 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/280d1cb0-534d-49b9-b267-7882dc079daa-console-config\") pod \"console-75c69c9dc9-vl49s\" (UID: \"280d1cb0-534d-49b9-b267-7882dc079daa\") " pod="openshift-console/console-75c69c9dc9-vl49s" Jul 20 21:07:24.062693 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:24.062609 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/280d1cb0-534d-49b9-b267-7882dc079daa-oauth-serving-cert\") pod \"console-75c69c9dc9-vl49s\" (UID: \"280d1cb0-534d-49b9-b267-7882dc079daa\") " pod="openshift-console/console-75c69c9dc9-vl49s" Jul 20 21:07:24.062693 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:24.062682 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/280d1cb0-534d-49b9-b267-7882dc079daa-console-oauth-config\") pod \"console-75c69c9dc9-vl49s\" (UID: \"280d1cb0-534d-49b9-b267-7882dc079daa\") " pod="openshift-console/console-75c69c9dc9-vl49s" Jul 20 21:07:24.063293 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:24.063238 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/280d1cb0-534d-49b9-b267-7882dc079daa-service-ca\") pod \"console-75c69c9dc9-vl49s\" (UID: \"280d1cb0-534d-49b9-b267-7882dc079daa\") " pod="openshift-console/console-75c69c9dc9-vl49s" Jul 20 21:07:24.063435 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:24.063338 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/280d1cb0-534d-49b9-b267-7882dc079daa-console-config\") pod \"console-75c69c9dc9-vl49s\" (UID: \"280d1cb0-534d-49b9-b267-7882dc079daa\") " pod="openshift-console/console-75c69c9dc9-vl49s" Jul 20 21:07:24.063435 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:24.063396 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/280d1cb0-534d-49b9-b267-7882dc079daa-oauth-serving-cert\") pod \"console-75c69c9dc9-vl49s\" (UID: \"280d1cb0-534d-49b9-b267-7882dc079daa\") " pod="openshift-console/console-75c69c9dc9-vl49s" Jul 20 21:07:24.065620 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:24.065599 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/280d1cb0-534d-49b9-b267-7882dc079daa-console-serving-cert\") pod \"console-75c69c9dc9-vl49s\" (UID: \"280d1cb0-534d-49b9-b267-7882dc079daa\") " pod="openshift-console/console-75c69c9dc9-vl49s" Jul 20 21:07:24.066278 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:24.066229 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/280d1cb0-534d-49b9-b267-7882dc079daa-console-oauth-config\") pod \"console-75c69c9dc9-vl49s\" (UID: \"280d1cb0-534d-49b9-b267-7882dc079daa\") " pod="openshift-console/console-75c69c9dc9-vl49s" Jul 20 21:07:24.071125 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:24.071101 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g94n\" (UniqueName: \"kubernetes.io/projected/280d1cb0-534d-49b9-b267-7882dc079daa-kube-api-access-6g94n\") pod \"console-75c69c9dc9-vl49s\" (UID: \"280d1cb0-534d-49b9-b267-7882dc079daa\") " pod="openshift-console/console-75c69c9dc9-vl49s" Jul 20 21:07:24.209387 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:24.209302 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75c69c9dc9-vl49s" Jul 20 21:07:24.425149 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:24.425109 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75c69c9dc9-vl49s"] Jul 20 21:07:24.428108 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:07:24.428077 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod280d1cb0_534d_49b9_b267_7882dc079daa.slice/crio-a65a60a0e5ba5df2ee2580723913c2102a7afe969d25db5b5686673a4ddcb2da WatchSource:0}: Error finding container a65a60a0e5ba5df2ee2580723913c2102a7afe969d25db5b5686673a4ddcb2da: Status 404 returned error can't find the container with id a65a60a0e5ba5df2ee2580723913c2102a7afe969d25db5b5686673a4ddcb2da Jul 20 21:07:25.073178 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:25.073122 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rq8g2" event={"ID":"399a1523-41ed-4080-9321-66533820eded","Type":"ContainerStarted","Data":"78b0439fc56b6e6079cc0af2d8045ea77f566bdaffad0736a57609f3d477e993"} Jul 20 21:07:25.074319 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:25.074296 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75c69c9dc9-vl49s" event={"ID":"280d1cb0-534d-49b9-b267-7882dc079daa","Type":"ContainerStarted","Data":"a65a60a0e5ba5df2ee2580723913c2102a7afe969d25db5b5686673a4ddcb2da"} Jul 20 21:07:25.092364 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:25.092323 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rq8g2" podStartSLOduration=18.406336126 podStartE2EDuration="30.092310038s" totalImagesPullingTime="11.685973912s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 21:06:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:07:25.090825862 +0000 UTC m=+284.416528805" watchObservedRunningTime="2026-07-20 21:07:25.092310038 +0000 UTC m=+284.418012981" Jul 20 21:07:25.305007 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:25.304967 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:25.976353 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:25.976314 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cc5fdb7-149c-47f7-805a-cadc7c5f2599-metrics-tls\") pod \"dns-default-zp575\" (UID: \"7cc5fdb7-149c-47f7-805a-cadc7c5f2599\") " pod="openshift-dns/dns-default-zp575" Jul 20 21:07:25.978611 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:25.978588 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cc5fdb7-149c-47f7-805a-cadc7c5f2599-metrics-tls\") pod \"dns-default-zp575\" (UID: \"7cc5fdb7-149c-47f7-805a-cadc7c5f2599\") " pod="openshift-dns/dns-default-zp575" Jul 20 21:07:26.065976 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:26.065941 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-n29b9\"" Jul 20 21:07:26.074242 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:26.074215 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zp575" Jul 20 21:07:26.215247 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:26.215213 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zp575"] Jul 20 21:07:26.218059 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:07:26.218035 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cc5fdb7_149c_47f7_805a_cadc7c5f2599.slice/crio-353cd4225d3664487302223345c236fc70ff4a860e2d726147dd6fd314cd984d WatchSource:0}: Error finding container 353cd4225d3664487302223345c236fc70ff4a860e2d726147dd6fd314cd984d: Status 404 returned error can't find the container with id 353cd4225d3664487302223345c236fc70ff4a860e2d726147dd6fd314cd984d Jul 20 21:07:27.082873 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:27.082833 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zp575" event={"ID":"7cc5fdb7-149c-47f7-805a-cadc7c5f2599","Type":"ContainerStarted","Data":"353cd4225d3664487302223345c236fc70ff4a860e2d726147dd6fd314cd984d"} Jul 20 21:07:27.157150 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:27.157102 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:27.803816 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:27.803742 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:27.904229 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:27.904187 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:28.014981 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:28.014954 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:28.089010 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:28.088974 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75c69c9dc9-vl49s" event={"ID":"280d1cb0-534d-49b9-b267-7882dc079daa","Type":"ContainerStarted","Data":"14691d1ad9177fab64de55ce9203ad998575b9f0b222c99d36c982a4f94f7953"} Jul 20 21:07:28.107491 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:28.107441 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-75c69c9dc9-vl49s" podStartSLOduration=1.832571618 podStartE2EDuration="5.107418656s" totalImagesPullingTime="3.274847038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 21:07:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:07:28.105849748 +0000 UTC m=+287.431552694" watchObservedRunningTime="2026-07-20 21:07:28.107418656 +0000 UTC m=+287.433121647" Jul 20 21:07:28.109812 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:28.109782 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:28.174660 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:28.174623 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:28.872841 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:28.872747 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:29.093936 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.093902 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zp575" event={"ID":"7cc5fdb7-149c-47f7-805a-cadc7c5f2599","Type":"ContainerStarted","Data":"6db98c19df5f79a5a636fa5ef1020f8b06861dd9165220dfb589628439199b66"} Jul 20 21:07:29.094385 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.093955 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zp575" event={"ID":"7cc5fdb7-149c-47f7-805a-cadc7c5f2599","Type":"ContainerStarted","Data":"e1e96d46140104a613c1b8c28ffe728137e64ed58ffe683713414fb9bc5d0b40"} Jul 20 21:07:29.112253 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.112182 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:29.117181 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.117124 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-zp575" podStartSLOduration=250.485354808 podStartE2EDuration="4m13.117110591s" totalImagesPullingTime="2.631755783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 21:03:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:07:29.116356236 +0000 UTC m=+288.442059178" watchObservedRunningTime="2026-07-20 21:07:29.117110591 +0000 UTC m=+288.442813534" Jul 20 21:07:29.294773 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.294736 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:29.363649 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.363571 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-rs8r9"] Jul 20 21:07:29.366811 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.366794 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-rs8r9" Jul 20 21:07:29.370051 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.370024 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 20 21:07:29.371355 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.371333 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-gsq87\"" Jul 20 21:07:29.371479 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.371358 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 20 21:07:29.371479 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.371365 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 20 21:07:29.371479 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.371369 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 20 21:07:29.382555 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.382533 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-rs8r9"] Jul 20 21:07:29.393603 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.393573 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-s45dv"] Jul 20 21:07:29.396617 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.396600 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s45dv" Jul 20 21:07:29.399846 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.399799 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 20 21:07:29.399989 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.399968 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 20 21:07:29.400207 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.400185 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 20 21:07:29.400319 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.400219 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8pvq7\"" Jul 20 21:07:29.400388 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.400359 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:29.405705 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.405680 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/700afcf1-b8cd-43d7-9c3b-e77e048bc338-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-rs8r9\" (UID: \"700afcf1-b8cd-43d7-9c3b-e77e048bc338\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rs8r9" Jul 20 21:07:29.405821 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.405732 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/700afcf1-b8cd-43d7-9c3b-e77e048bc338-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-rs8r9\" (UID: \"700afcf1-b8cd-43d7-9c3b-e77e048bc338\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rs8r9" Jul 20 21:07:29.405821 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.405789 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/700afcf1-b8cd-43d7-9c3b-e77e048bc338-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-rs8r9\" (UID: \"700afcf1-b8cd-43d7-9c3b-e77e048bc338\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rs8r9" Jul 20 21:07:29.405947 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.405848 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/700afcf1-b8cd-43d7-9c3b-e77e048bc338-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-rs8r9\" (UID: \"700afcf1-b8cd-43d7-9c3b-e77e048bc338\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rs8r9" Jul 20 21:07:29.405997 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.405940 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/700afcf1-b8cd-43d7-9c3b-e77e048bc338-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-rs8r9\" (UID: \"700afcf1-b8cd-43d7-9c3b-e77e048bc338\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rs8r9" Jul 20 21:07:29.406047 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.405998 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phv6s\" (UniqueName: \"kubernetes.io/projected/700afcf1-b8cd-43d7-9c3b-e77e048bc338-kube-api-access-phv6s\") pod \"kube-state-metrics-56c5594674-rs8r9\" (UID: \"700afcf1-b8cd-43d7-9c3b-e77e048bc338\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rs8r9" Jul 20 21:07:29.482795 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.482761 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:29.506561 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.506531 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2be85c96-229e-4710-b94b-1b98e7b1217c-node-exporter-accelerators-collector-config\") pod \"node-exporter-s45dv\" (UID: \"2be85c96-229e-4710-b94b-1b98e7b1217c\") " pod="openshift-monitoring/node-exporter-s45dv" Jul 20 21:07:29.506709 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.506564 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl5qc\" (UniqueName: \"kubernetes.io/projected/2be85c96-229e-4710-b94b-1b98e7b1217c-kube-api-access-dl5qc\") pod \"node-exporter-s45dv\" (UID: \"2be85c96-229e-4710-b94b-1b98e7b1217c\") " pod="openshift-monitoring/node-exporter-s45dv" Jul 20 21:07:29.506709 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.506586 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phv6s\" (UniqueName: \"kubernetes.io/projected/700afcf1-b8cd-43d7-9c3b-e77e048bc338-kube-api-access-phv6s\") pod \"kube-state-metrics-56c5594674-rs8r9\" (UID: \"700afcf1-b8cd-43d7-9c3b-e77e048bc338\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rs8r9" Jul 20 21:07:29.506709 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.506611 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2be85c96-229e-4710-b94b-1b98e7b1217c-metrics-client-ca\") pod \"node-exporter-s45dv\" (UID: \"2be85c96-229e-4710-b94b-1b98e7b1217c\") " pod="openshift-monitoring/node-exporter-s45dv" Jul 20 21:07:29.506709 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.506646 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2be85c96-229e-4710-b94b-1b98e7b1217c-sys\") pod \"node-exporter-s45dv\" (UID: \"2be85c96-229e-4710-b94b-1b98e7b1217c\") " pod="openshift-monitoring/node-exporter-s45dv" Jul 20 21:07:29.506709 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.506676 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2be85c96-229e-4710-b94b-1b98e7b1217c-node-exporter-tls\") pod \"node-exporter-s45dv\" (UID: \"2be85c96-229e-4710-b94b-1b98e7b1217c\") " pod="openshift-monitoring/node-exporter-s45dv" Jul 20 21:07:29.506709 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.506709 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2be85c96-229e-4710-b94b-1b98e7b1217c-node-exporter-textfile\") pod \"node-exporter-s45dv\" (UID: \"2be85c96-229e-4710-b94b-1b98e7b1217c\") " pod="openshift-monitoring/node-exporter-s45dv" Jul 20 21:07:29.506983 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.506733 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2be85c96-229e-4710-b94b-1b98e7b1217c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s45dv\" (UID: \"2be85c96-229e-4710-b94b-1b98e7b1217c\") " pod="openshift-monitoring/node-exporter-s45dv" Jul 20 21:07:29.506983 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.506756 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/700afcf1-b8cd-43d7-9c3b-e77e048bc338-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-rs8r9\" (UID: \"700afcf1-b8cd-43d7-9c3b-e77e048bc338\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rs8r9" Jul 20 21:07:29.506983 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.506780 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/700afcf1-b8cd-43d7-9c3b-e77e048bc338-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-rs8r9\" (UID: \"700afcf1-b8cd-43d7-9c3b-e77e048bc338\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rs8r9" Jul 20 21:07:29.506983 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.506807 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/700afcf1-b8cd-43d7-9c3b-e77e048bc338-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-rs8r9\" (UID: \"700afcf1-b8cd-43d7-9c3b-e77e048bc338\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rs8r9" Jul 20 21:07:29.506983 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.506833 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/700afcf1-b8cd-43d7-9c3b-e77e048bc338-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-rs8r9\" (UID: \"700afcf1-b8cd-43d7-9c3b-e77e048bc338\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rs8r9" Jul 20 21:07:29.506983 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:07:29.506885 2562 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 20 21:07:29.506983 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.506941 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2be85c96-229e-4710-b94b-1b98e7b1217c-node-exporter-wtmp\") pod \"node-exporter-s45dv\" (UID: \"2be85c96-229e-4710-b94b-1b98e7b1217c\") " pod="openshift-monitoring/node-exporter-s45dv" Jul 20 21:07:29.506983 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:07:29.506950 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/700afcf1-b8cd-43d7-9c3b-e77e048bc338-kube-state-metrics-tls podName:700afcf1-b8cd-43d7-9c3b-e77e048bc338 nodeName:}" failed. No retries permitted until 2026-07-20 21:07:30.00692787 +0000 UTC m=+289.332630792 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/700afcf1-b8cd-43d7-9c3b-e77e048bc338-kube-state-metrics-tls") pod "kube-state-metrics-56c5594674-rs8r9" (UID: "700afcf1-b8cd-43d7-9c3b-e77e048bc338") : secret "kube-state-metrics-tls" not found Jul 20 21:07:29.507308 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.507008 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2be85c96-229e-4710-b94b-1b98e7b1217c-root\") pod \"node-exporter-s45dv\" (UID: \"2be85c96-229e-4710-b94b-1b98e7b1217c\") " pod="openshift-monitoring/node-exporter-s45dv" Jul 20 21:07:29.507308 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.507052 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/700afcf1-b8cd-43d7-9c3b-e77e048bc338-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-rs8r9\" (UID: \"700afcf1-b8cd-43d7-9c3b-e77e048bc338\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rs8r9" Jul 20 21:07:29.507402 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.507382 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/700afcf1-b8cd-43d7-9c3b-e77e048bc338-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-rs8r9\" (UID: \"700afcf1-b8cd-43d7-9c3b-e77e048bc338\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rs8r9" Jul 20 21:07:29.507474 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.507456 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/700afcf1-b8cd-43d7-9c3b-e77e048bc338-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-rs8r9\" (UID: \"700afcf1-b8cd-43d7-9c3b-e77e048bc338\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rs8r9" Jul 20 21:07:29.507532 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.507495 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/700afcf1-b8cd-43d7-9c3b-e77e048bc338-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-rs8r9\" (UID: \"700afcf1-b8cd-43d7-9c3b-e77e048bc338\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rs8r9" Jul 20 21:07:29.509249 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.509227 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/700afcf1-b8cd-43d7-9c3b-e77e048bc338-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-rs8r9\" (UID: \"700afcf1-b8cd-43d7-9c3b-e77e048bc338\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rs8r9" Jul 20 21:07:29.515083 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.515064 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phv6s\" (UniqueName: \"kubernetes.io/projected/700afcf1-b8cd-43d7-9c3b-e77e048bc338-kube-api-access-phv6s\") pod \"kube-state-metrics-56c5594674-rs8r9\" (UID: \"700afcf1-b8cd-43d7-9c3b-e77e048bc338\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rs8r9" Jul 20 21:07:29.580549 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.580501 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:29.607438 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.607401 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2be85c96-229e-4710-b94b-1b98e7b1217c-node-exporter-wtmp\") pod \"node-exporter-s45dv\" (UID: \"2be85c96-229e-4710-b94b-1b98e7b1217c\") " pod="openshift-monitoring/node-exporter-s45dv" Jul 20 21:07:29.607609 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.607447 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2be85c96-229e-4710-b94b-1b98e7b1217c-root\") pod \"node-exporter-s45dv\" (UID: \"2be85c96-229e-4710-b94b-1b98e7b1217c\") " pod="openshift-monitoring/node-exporter-s45dv" Jul 20 21:07:29.607609 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.607490 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2be85c96-229e-4710-b94b-1b98e7b1217c-node-exporter-accelerators-collector-config\") pod \"node-exporter-s45dv\" (UID: \"2be85c96-229e-4710-b94b-1b98e7b1217c\") " pod="openshift-monitoring/node-exporter-s45dv" Jul 20 21:07:29.607609 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.607509 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl5qc\" (UniqueName: \"kubernetes.io/projected/2be85c96-229e-4710-b94b-1b98e7b1217c-kube-api-access-dl5qc\") pod \"node-exporter-s45dv\" (UID: \"2be85c96-229e-4710-b94b-1b98e7b1217c\") " pod="openshift-monitoring/node-exporter-s45dv" Jul 20 21:07:29.607609 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.607533 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2be85c96-229e-4710-b94b-1b98e7b1217c-metrics-client-ca\") pod \"node-exporter-s45dv\" (UID: \"2be85c96-229e-4710-b94b-1b98e7b1217c\") " pod="openshift-monitoring/node-exporter-s45dv" Jul 20 21:07:29.607609 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.607549 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2be85c96-229e-4710-b94b-1b98e7b1217c-sys\") pod \"node-exporter-s45dv\" (UID: \"2be85c96-229e-4710-b94b-1b98e7b1217c\") " pod="openshift-monitoring/node-exporter-s45dv" Jul 20 21:07:29.607609 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.607567 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2be85c96-229e-4710-b94b-1b98e7b1217c-node-exporter-tls\") pod \"node-exporter-s45dv\" (UID: \"2be85c96-229e-4710-b94b-1b98e7b1217c\") " pod="openshift-monitoring/node-exporter-s45dv" Jul 20 21:07:29.607609 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.607589 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2be85c96-229e-4710-b94b-1b98e7b1217c-node-exporter-wtmp\") pod \"node-exporter-s45dv\" (UID: \"2be85c96-229e-4710-b94b-1b98e7b1217c\") " pod="openshift-monitoring/node-exporter-s45dv" Jul 20 21:07:29.607609 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.607591 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2be85c96-229e-4710-b94b-1b98e7b1217c-root\") pod \"node-exporter-s45dv\" (UID: \"2be85c96-229e-4710-b94b-1b98e7b1217c\") " pod="openshift-monitoring/node-exporter-s45dv" Jul 20 21:07:29.607964 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.607601 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2be85c96-229e-4710-b94b-1b98e7b1217c-node-exporter-textfile\") pod \"node-exporter-s45dv\" (UID: \"2be85c96-229e-4710-b94b-1b98e7b1217c\") " pod="openshift-monitoring/node-exporter-s45dv" Jul 20 21:07:29.607964 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.607645 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2be85c96-229e-4710-b94b-1b98e7b1217c-sys\") pod \"node-exporter-s45dv\" (UID: \"2be85c96-229e-4710-b94b-1b98e7b1217c\") " pod="openshift-monitoring/node-exporter-s45dv" Jul 20 21:07:29.607964 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.607711 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2be85c96-229e-4710-b94b-1b98e7b1217c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s45dv\" (UID: \"2be85c96-229e-4710-b94b-1b98e7b1217c\") " pod="openshift-monitoring/node-exporter-s45dv" Jul 20 21:07:29.607964 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.607905 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2be85c96-229e-4710-b94b-1b98e7b1217c-node-exporter-textfile\") pod \"node-exporter-s45dv\" (UID: \"2be85c96-229e-4710-b94b-1b98e7b1217c\") " pod="openshift-monitoring/node-exporter-s45dv" Jul 20 21:07:29.608240 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.608220 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2be85c96-229e-4710-b94b-1b98e7b1217c-node-exporter-accelerators-collector-config\") pod \"node-exporter-s45dv\" (UID: \"2be85c96-229e-4710-b94b-1b98e7b1217c\") " pod="openshift-monitoring/node-exporter-s45dv" Jul 20 21:07:29.608313 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.608224 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2be85c96-229e-4710-b94b-1b98e7b1217c-metrics-client-ca\") pod \"node-exporter-s45dv\" (UID: \"2be85c96-229e-4710-b94b-1b98e7b1217c\") " pod="openshift-monitoring/node-exporter-s45dv" Jul 20 21:07:29.609859 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.609841 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2be85c96-229e-4710-b94b-1b98e7b1217c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s45dv\" (UID: \"2be85c96-229e-4710-b94b-1b98e7b1217c\") " pod="openshift-monitoring/node-exporter-s45dv" Jul 20 21:07:29.610056 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.610038 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2be85c96-229e-4710-b94b-1b98e7b1217c-node-exporter-tls\") pod \"node-exporter-s45dv\" (UID: \"2be85c96-229e-4710-b94b-1b98e7b1217c\") " pod="openshift-monitoring/node-exporter-s45dv" Jul 20 21:07:29.616596 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.616545 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl5qc\" (UniqueName: \"kubernetes.io/projected/2be85c96-229e-4710-b94b-1b98e7b1217c-kube-api-access-dl5qc\") pod \"node-exporter-s45dv\" (UID: \"2be85c96-229e-4710-b94b-1b98e7b1217c\") " pod="openshift-monitoring/node-exporter-s45dv" Jul 20 21:07:29.681182 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.681148 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:29.706326 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.706295 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s45dv" Jul 20 21:07:29.717891 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:07:29.717860 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2be85c96_229e_4710_b94b_1b98e7b1217c.slice/crio-3c103cebc430d22fe612063f14961bb92b89470ca073d01fb55c0a7384a27cf8 WatchSource:0}: Error finding container 3c103cebc430d22fe612063f14961bb92b89470ca073d01fb55c0a7384a27cf8: Status 404 returned error can't find the container with id 3c103cebc430d22fe612063f14961bb92b89470ca073d01fb55c0a7384a27cf8 Jul 20 21:07:29.804467 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:29.804414 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:30.010750 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:30.010704 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/700afcf1-b8cd-43d7-9c3b-e77e048bc338-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-rs8r9\" (UID: \"700afcf1-b8cd-43d7-9c3b-e77e048bc338\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rs8r9" Jul 20 21:07:30.017020 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:30.013424 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/700afcf1-b8cd-43d7-9c3b-e77e048bc338-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-rs8r9\" (UID: \"700afcf1-b8cd-43d7-9c3b-e77e048bc338\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rs8r9" Jul 20 21:07:30.020198 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:30.020166 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:30.098353 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:30.098303 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s45dv" event={"ID":"2be85c96-229e-4710-b94b-1b98e7b1217c","Type":"ContainerStarted","Data":"3c103cebc430d22fe612063f14961bb92b89470ca073d01fb55c0a7384a27cf8"} Jul 20 21:07:30.098772 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:30.098483 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zp575" Jul 20 21:07:30.102000 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:30.101978 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5748dd95bb-qhfpf" Jul 20 21:07:30.122696 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:30.122663 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:30.276127 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:30.276048 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-rs8r9" Jul 20 21:07:30.331846 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:30.331705 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:30.420193 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:30.420118 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:30.449226 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:30.449190 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-rs8r9"] Jul 20 21:07:30.453245 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:07:30.453214 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod700afcf1_b8cd_43d7_9c3b_e77e048bc338.slice/crio-7dc3971dd13733e325c5324b4646449fce7388c5636468372002ff73d1c2d2cb WatchSource:0}: Error finding container 7dc3971dd13733e325c5324b4646449fce7388c5636468372002ff73d1c2d2cb: Status 404 returned error can't find the container with id 7dc3971dd13733e325c5324b4646449fce7388c5636468372002ff73d1c2d2cb Jul 20 21:07:30.740351 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:30.740271 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:31.105764 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:31.104683 2562 generic.go:356] "Generic (PLEG): container finished" podID="2be85c96-229e-4710-b94b-1b98e7b1217c" containerID="86ca74825df77bc990b3be475deec6913b6627e4ce5ebc08bff9713709c44c57" exitCode=0 Jul 20 21:07:31.105764 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:31.104813 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s45dv" event={"ID":"2be85c96-229e-4710-b94b-1b98e7b1217c","Type":"ContainerDied","Data":"86ca74825df77bc990b3be475deec6913b6627e4ce5ebc08bff9713709c44c57"} Jul 20 21:07:31.107719 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:31.107689 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-rs8r9" event={"ID":"700afcf1-b8cd-43d7-9c3b-e77e048bc338","Type":"ContainerStarted","Data":"7dc3971dd13733e325c5324b4646449fce7388c5636468372002ff73d1c2d2cb"} Jul 20 21:07:31.343465 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:31.343434 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5c8d56f8b8-hxvgr"] Jul 20 21:07:31.346970 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:31.346952 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5c8d56f8b8-hxvgr" Jul 20 21:07:31.349514 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:31.349488 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 20 21:07:31.349684 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:31.349574 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-dbv1dfrmnc94m\"" Jul 20 21:07:31.349684 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:31.349637 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 20 21:07:31.349684 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:31.349671 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 20 21:07:31.349876 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:31.349721 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 20 21:07:31.349876 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:31.349735 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 20 21:07:31.349876 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:31.349744 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-7s77f\"" Jul 20 21:07:31.359286 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:31.359231 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5c8d56f8b8-hxvgr"] Jul 20 21:07:31.423271 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:31.423230 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ff6806f9-668b-4512-906a-df4a69b73ba9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5c8d56f8b8-hxvgr\" (UID: \"ff6806f9-668b-4512-906a-df4a69b73ba9\") " pod="openshift-monitoring/thanos-querier-5c8d56f8b8-hxvgr" Jul 20 21:07:31.423471 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:31.423299 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ff6806f9-668b-4512-906a-df4a69b73ba9-secret-thanos-querier-tls\") pod \"thanos-querier-5c8d56f8b8-hxvgr\" (UID: \"ff6806f9-668b-4512-906a-df4a69b73ba9\") " pod="openshift-monitoring/thanos-querier-5c8d56f8b8-hxvgr" Jul 20 21:07:31.423471 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:31.423331 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff6806f9-668b-4512-906a-df4a69b73ba9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5c8d56f8b8-hxvgr\" (UID: \"ff6806f9-668b-4512-906a-df4a69b73ba9\") " pod="openshift-monitoring/thanos-querier-5c8d56f8b8-hxvgr" Jul 20 21:07:31.423471 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:31.423368 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff6806f9-668b-4512-906a-df4a69b73ba9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5c8d56f8b8-hxvgr\" (UID: \"ff6806f9-668b-4512-906a-df4a69b73ba9\") " pod="openshift-monitoring/thanos-querier-5c8d56f8b8-hxvgr" Jul 20 21:07:31.423471 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:31.423435 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff6806f9-668b-4512-906a-df4a69b73ba9-metrics-client-ca\") pod \"thanos-querier-5c8d56f8b8-hxvgr\" (UID: \"ff6806f9-668b-4512-906a-df4a69b73ba9\") " pod="openshift-monitoring/thanos-querier-5c8d56f8b8-hxvgr" Jul 20 21:07:31.423626 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:31.423475 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ff6806f9-668b-4512-906a-df4a69b73ba9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5c8d56f8b8-hxvgr\" (UID: \"ff6806f9-668b-4512-906a-df4a69b73ba9\") " pod="openshift-monitoring/thanos-querier-5c8d56f8b8-hxvgr" Jul 20 21:07:31.423626 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:31.423501 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnhww\" (UniqueName: \"kubernetes.io/projected/ff6806f9-668b-4512-906a-df4a69b73ba9-kube-api-access-mnhww\") pod \"thanos-querier-5c8d56f8b8-hxvgr\" (UID: \"ff6806f9-668b-4512-906a-df4a69b73ba9\") " pod="openshift-monitoring/thanos-querier-5c8d56f8b8-hxvgr" Jul 20 21:07:31.423626 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:31.423532 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ff6806f9-668b-4512-906a-df4a69b73ba9-secret-grpc-tls\") pod \"thanos-querier-5c8d56f8b8-hxvgr\" (UID: \"ff6806f9-668b-4512-906a-df4a69b73ba9\") " pod="openshift-monitoring/thanos-querier-5c8d56f8b8-hxvgr" Jul 20 21:07:31.524936 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:31.524889 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff6806f9-668b-4512-906a-df4a69b73ba9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5c8d56f8b8-hxvgr\" (UID: \"ff6806f9-668b-4512-906a-df4a69b73ba9\") " pod="openshift-monitoring/thanos-querier-5c8d56f8b8-hxvgr" Jul 20 21:07:31.525128 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:31.524945 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff6806f9-668b-4512-906a-df4a69b73ba9-metrics-client-ca\") pod \"thanos-querier-5c8d56f8b8-hxvgr\" (UID: \"ff6806f9-668b-4512-906a-df4a69b73ba9\") " pod="openshift-monitoring/thanos-querier-5c8d56f8b8-hxvgr" Jul 20 21:07:31.525128 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:31.524972 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ff6806f9-668b-4512-906a-df4a69b73ba9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5c8d56f8b8-hxvgr\" (UID: \"ff6806f9-668b-4512-906a-df4a69b73ba9\") " pod="openshift-monitoring/thanos-querier-5c8d56f8b8-hxvgr" Jul 20 21:07:31.525128 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:31.525003 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnhww\" (UniqueName: \"kubernetes.io/projected/ff6806f9-668b-4512-906a-df4a69b73ba9-kube-api-access-mnhww\") pod \"thanos-querier-5c8d56f8b8-hxvgr\" (UID: \"ff6806f9-668b-4512-906a-df4a69b73ba9\") " pod="openshift-monitoring/thanos-querier-5c8d56f8b8-hxvgr" Jul 20 21:07:31.525128 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:31.525034 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ff6806f9-668b-4512-906a-df4a69b73ba9-secret-grpc-tls\") pod \"thanos-querier-5c8d56f8b8-hxvgr\" (UID: \"ff6806f9-668b-4512-906a-df4a69b73ba9\") " pod="openshift-monitoring/thanos-querier-5c8d56f8b8-hxvgr" Jul 20 21:07:31.525128 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:31.525103 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ff6806f9-668b-4512-906a-df4a69b73ba9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5c8d56f8b8-hxvgr\" (UID: \"ff6806f9-668b-4512-906a-df4a69b73ba9\") " pod="openshift-monitoring/thanos-querier-5c8d56f8b8-hxvgr" Jul 20 21:07:31.525403 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:31.525164 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ff6806f9-668b-4512-906a-df4a69b73ba9-secret-thanos-querier-tls\") pod \"thanos-querier-5c8d56f8b8-hxvgr\" (UID: \"ff6806f9-668b-4512-906a-df4a69b73ba9\") " pod="openshift-monitoring/thanos-querier-5c8d56f8b8-hxvgr" Jul 20 21:07:31.525403 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:31.525181 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff6806f9-668b-4512-906a-df4a69b73ba9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5c8d56f8b8-hxvgr\" (UID: \"ff6806f9-668b-4512-906a-df4a69b73ba9\") " pod="openshift-monitoring/thanos-querier-5c8d56f8b8-hxvgr" Jul 20 21:07:31.529725 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:31.529700 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ff6806f9-668b-4512-906a-df4a69b73ba9-secret-thanos-querier-tls\") pod \"thanos-querier-5c8d56f8b8-hxvgr\" (UID: \"ff6806f9-668b-4512-906a-df4a69b73ba9\") " pod="openshift-monitoring/thanos-querier-5c8d56f8b8-hxvgr" Jul 20 21:07:31.531016 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:31.530986 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ff6806f9-668b-4512-906a-df4a69b73ba9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5c8d56f8b8-hxvgr\" (UID: \"ff6806f9-668b-4512-906a-df4a69b73ba9\") " pod="openshift-monitoring/thanos-querier-5c8d56f8b8-hxvgr" Jul 20 21:07:31.531118 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:31.531064 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff6806f9-668b-4512-906a-df4a69b73ba9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5c8d56f8b8-hxvgr\" (UID: \"ff6806f9-668b-4512-906a-df4a69b73ba9\") " pod="openshift-monitoring/thanos-querier-5c8d56f8b8-hxvgr" Jul 20 21:07:31.532717 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:31.532688 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff6806f9-668b-4512-906a-df4a69b73ba9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5c8d56f8b8-hxvgr\" (UID: \"ff6806f9-668b-4512-906a-df4a69b73ba9\") " pod="openshift-monitoring/thanos-querier-5c8d56f8b8-hxvgr" Jul 20 21:07:31.533065 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:31.533046 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff6806f9-668b-4512-906a-df4a69b73ba9-metrics-client-ca\") pod \"thanos-querier-5c8d56f8b8-hxvgr\" (UID: \"ff6806f9-668b-4512-906a-df4a69b73ba9\") " pod="openshift-monitoring/thanos-querier-5c8d56f8b8-hxvgr" Jul 20 21:07:31.534540 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:31.534519 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ff6806f9-668b-4512-906a-df4a69b73ba9-secret-grpc-tls\") pod \"thanos-querier-5c8d56f8b8-hxvgr\" (UID: \"ff6806f9-668b-4512-906a-df4a69b73ba9\") " pod="openshift-monitoring/thanos-querier-5c8d56f8b8-hxvgr" Jul 20 21:07:31.535702 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:31.535677 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ff6806f9-668b-4512-906a-df4a69b73ba9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5c8d56f8b8-hxvgr\" (UID: \"ff6806f9-668b-4512-906a-df4a69b73ba9\") " pod="openshift-monitoring/thanos-querier-5c8d56f8b8-hxvgr" Jul 20 21:07:31.536049 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:31.536030 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnhww\" (UniqueName: \"kubernetes.io/projected/ff6806f9-668b-4512-906a-df4a69b73ba9-kube-api-access-mnhww\") pod \"thanos-querier-5c8d56f8b8-hxvgr\" (UID: \"ff6806f9-668b-4512-906a-df4a69b73ba9\") " pod="openshift-monitoring/thanos-querier-5c8d56f8b8-hxvgr" Jul 20 21:07:31.656784 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:31.656694 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5c8d56f8b8-hxvgr" Jul 20 21:07:31.804409 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:31.804380 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5c8d56f8b8-hxvgr"] Jul 20 21:07:31.806614 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:07:31.806583 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff6806f9_668b_4512_906a_df4a69b73ba9.slice/crio-b400da04035fadd8a03ea4bed37b8273e582e8de308e17d81bd56510841f50fa WatchSource:0}: Error finding container b400da04035fadd8a03ea4bed37b8273e582e8de308e17d81bd56510841f50fa: Status 404 returned error can't find the container with id b400da04035fadd8a03ea4bed37b8273e582e8de308e17d81bd56510841f50fa Jul 20 21:07:32.118318 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:32.118278 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s45dv" event={"ID":"2be85c96-229e-4710-b94b-1b98e7b1217c","Type":"ContainerStarted","Data":"c7b080bdb4a32ea1880eb9dc5b84cf74a285f0cb089f79d129512a24894e3839"} Jul 20 21:07:32.118805 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:32.118395 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s45dv" event={"ID":"2be85c96-229e-4710-b94b-1b98e7b1217c","Type":"ContainerStarted","Data":"2d594ab5fd0f355a37350dcc191f4a03283e19b4e966cb70aa4750bb0556583c"} Jul 20 21:07:32.119606 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:32.119573 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c8d56f8b8-hxvgr" event={"ID":"ff6806f9-668b-4512-906a-df4a69b73ba9","Type":"ContainerStarted","Data":"b400da04035fadd8a03ea4bed37b8273e582e8de308e17d81bd56510841f50fa"} Jul 20 21:07:32.142911 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:32.142516 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-s45dv" podStartSLOduration=2.023940127 podStartE2EDuration="3.142496247s" totalImagesPullingTime="1.053703228s" totalInitContainerRuntime="64.852892ms" isStatefulPod=true podCreationTimestamp="2026-07-20 21:07:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:07:32.14170809 +0000 UTC m=+291.467411034" watchObservedRunningTime="2026-07-20 21:07:32.142496247 +0000 UTC m=+291.468199192" Jul 20 21:07:32.284174 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:32.284125 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:32.384975 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:32.384948 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:32.659548 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:32.659423 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:32.744846 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:32.744813 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:32.806518 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:32.806460 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:33.070452 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.070401 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:33.124997 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.124965 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-rs8r9" event={"ID":"700afcf1-b8cd-43d7-9c3b-e77e048bc338","Type":"ContainerStarted","Data":"b0e58bf10b58757160521ab309ce2217ef4d430b98334c73505c07ef55d4e71b"} Jul 20 21:07:33.125402 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.125026 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-rs8r9" event={"ID":"700afcf1-b8cd-43d7-9c3b-e77e048bc338","Type":"ContainerStarted","Data":"e6326ec2ad59c481039255d322e727788f7da5b0bb14ecafa2f477fd6f3ee193"} Jul 20 21:07:33.125402 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.125050 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-rs8r9" event={"ID":"700afcf1-b8cd-43d7-9c3b-e77e048bc338","Type":"ContainerStarted","Data":"626bb724b8ea9b27703d3b0bfa336d7c509e8cef8de1c9d52a21ece0557dc24b"} Jul 20 21:07:33.147769 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.147708 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-rs8r9" podStartSLOduration=2.420569947 podStartE2EDuration="4.147692751s" totalImagesPullingTime="1.727122804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 21:07:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:07:33.146236823 +0000 UTC m=+292.471939769" watchObservedRunningTime="2026-07-20 21:07:33.147692751 +0000 UTC m=+292.473395693" Jul 20 21:07:33.155585 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.155559 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:33.271416 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.271387 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5bbdc6cd7f-c87gf"] Jul 20 21:07:33.274852 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.274829 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bbdc6cd7f-c87gf" Jul 20 21:07:33.283958 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.283933 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 20 21:07:33.290439 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.290416 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bbdc6cd7f-c87gf"] Jul 20 21:07:33.341338 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.341243 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4b57684d-9ed6-45f7-8584-da0b459485f5-console-oauth-config\") pod \"console-5bbdc6cd7f-c87gf\" (UID: \"4b57684d-9ed6-45f7-8584-da0b459485f5\") " pod="openshift-console/console-5bbdc6cd7f-c87gf" Jul 20 21:07:33.341338 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.341313 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4b57684d-9ed6-45f7-8584-da0b459485f5-console-config\") pod \"console-5bbdc6cd7f-c87gf\" (UID: \"4b57684d-9ed6-45f7-8584-da0b459485f5\") " pod="openshift-console/console-5bbdc6cd7f-c87gf" Jul 20 21:07:33.341563 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.341380 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4b57684d-9ed6-45f7-8584-da0b459485f5-console-serving-cert\") pod \"console-5bbdc6cd7f-c87gf\" (UID: \"4b57684d-9ed6-45f7-8584-da0b459485f5\") " pod="openshift-console/console-5bbdc6cd7f-c87gf" Jul 20 21:07:33.341563 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.341414 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b57684d-9ed6-45f7-8584-da0b459485f5-trusted-ca-bundle\") pod \"console-5bbdc6cd7f-c87gf\" (UID: \"4b57684d-9ed6-45f7-8584-da0b459485f5\") " pod="openshift-console/console-5bbdc6cd7f-c87gf" Jul 20 21:07:33.341563 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.341450 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4b57684d-9ed6-45f7-8584-da0b459485f5-oauth-serving-cert\") pod \"console-5bbdc6cd7f-c87gf\" (UID: \"4b57684d-9ed6-45f7-8584-da0b459485f5\") " pod="openshift-console/console-5bbdc6cd7f-c87gf" Jul 20 21:07:33.341563 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.341475 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4l9w\" (UniqueName: \"kubernetes.io/projected/4b57684d-9ed6-45f7-8584-da0b459485f5-kube-api-access-x4l9w\") pod \"console-5bbdc6cd7f-c87gf\" (UID: \"4b57684d-9ed6-45f7-8584-da0b459485f5\") " pod="openshift-console/console-5bbdc6cd7f-c87gf" Jul 20 21:07:33.341563 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.341508 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4b57684d-9ed6-45f7-8584-da0b459485f5-service-ca\") pod \"console-5bbdc6cd7f-c87gf\" (UID: \"4b57684d-9ed6-45f7-8584-da0b459485f5\") " pod="openshift-console/console-5bbdc6cd7f-c87gf" Jul 20 21:07:33.442327 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.442282 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4b57684d-9ed6-45f7-8584-da0b459485f5-console-oauth-config\") pod \"console-5bbdc6cd7f-c87gf\" (UID: \"4b57684d-9ed6-45f7-8584-da0b459485f5\") " pod="openshift-console/console-5bbdc6cd7f-c87gf" Jul 20 21:07:33.442491 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.442365 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4b57684d-9ed6-45f7-8584-da0b459485f5-console-config\") pod \"console-5bbdc6cd7f-c87gf\" (UID: \"4b57684d-9ed6-45f7-8584-da0b459485f5\") " pod="openshift-console/console-5bbdc6cd7f-c87gf" Jul 20 21:07:33.442491 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.442424 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4b57684d-9ed6-45f7-8584-da0b459485f5-console-serving-cert\") pod \"console-5bbdc6cd7f-c87gf\" (UID: \"4b57684d-9ed6-45f7-8584-da0b459485f5\") " pod="openshift-console/console-5bbdc6cd7f-c87gf" Jul 20 21:07:33.442491 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.442464 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b57684d-9ed6-45f7-8584-da0b459485f5-trusted-ca-bundle\") pod \"console-5bbdc6cd7f-c87gf\" (UID: \"4b57684d-9ed6-45f7-8584-da0b459485f5\") " pod="openshift-console/console-5bbdc6cd7f-c87gf" Jul 20 21:07:33.442660 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.442506 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4b57684d-9ed6-45f7-8584-da0b459485f5-oauth-serving-cert\") pod \"console-5bbdc6cd7f-c87gf\" (UID: \"4b57684d-9ed6-45f7-8584-da0b459485f5\") " pod="openshift-console/console-5bbdc6cd7f-c87gf" Jul 20 21:07:33.442660 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.442533 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4l9w\" (UniqueName: \"kubernetes.io/projected/4b57684d-9ed6-45f7-8584-da0b459485f5-kube-api-access-x4l9w\") pod \"console-5bbdc6cd7f-c87gf\" (UID: \"4b57684d-9ed6-45f7-8584-da0b459485f5\") " pod="openshift-console/console-5bbdc6cd7f-c87gf" Jul 20 21:07:33.442660 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.442573 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4b57684d-9ed6-45f7-8584-da0b459485f5-service-ca\") pod \"console-5bbdc6cd7f-c87gf\" (UID: \"4b57684d-9ed6-45f7-8584-da0b459485f5\") " pod="openshift-console/console-5bbdc6cd7f-c87gf" Jul 20 21:07:33.443292 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.443261 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4b57684d-9ed6-45f7-8584-da0b459485f5-console-config\") pod \"console-5bbdc6cd7f-c87gf\" (UID: \"4b57684d-9ed6-45f7-8584-da0b459485f5\") " pod="openshift-console/console-5bbdc6cd7f-c87gf" Jul 20 21:07:33.443447 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.443263 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4b57684d-9ed6-45f7-8584-da0b459485f5-oauth-serving-cert\") pod \"console-5bbdc6cd7f-c87gf\" (UID: \"4b57684d-9ed6-45f7-8584-da0b459485f5\") " pod="openshift-console/console-5bbdc6cd7f-c87gf" Jul 20 21:07:33.443877 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.443849 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4b57684d-9ed6-45f7-8584-da0b459485f5-service-ca\") pod \"console-5bbdc6cd7f-c87gf\" (UID: \"4b57684d-9ed6-45f7-8584-da0b459485f5\") " pod="openshift-console/console-5bbdc6cd7f-c87gf" Jul 20 21:07:33.444007 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.443890 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b57684d-9ed6-45f7-8584-da0b459485f5-trusted-ca-bundle\") pod \"console-5bbdc6cd7f-c87gf\" (UID: \"4b57684d-9ed6-45f7-8584-da0b459485f5\") " pod="openshift-console/console-5bbdc6cd7f-c87gf" Jul 20 21:07:33.445063 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.445042 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4b57684d-9ed6-45f7-8584-da0b459485f5-console-serving-cert\") pod \"console-5bbdc6cd7f-c87gf\" (UID: \"4b57684d-9ed6-45f7-8584-da0b459485f5\") " pod="openshift-console/console-5bbdc6cd7f-c87gf" Jul 20 21:07:33.445182 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.445102 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4b57684d-9ed6-45f7-8584-da0b459485f5-console-oauth-config\") pod \"console-5bbdc6cd7f-c87gf\" (UID: \"4b57684d-9ed6-45f7-8584-da0b459485f5\") " pod="openshift-console/console-5bbdc6cd7f-c87gf" Jul 20 21:07:33.453477 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.453456 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4l9w\" (UniqueName: \"kubernetes.io/projected/4b57684d-9ed6-45f7-8584-da0b459485f5-kube-api-access-x4l9w\") pod \"console-5bbdc6cd7f-c87gf\" (UID: \"4b57684d-9ed6-45f7-8584-da0b459485f5\") " pod="openshift-console/console-5bbdc6cd7f-c87gf" Jul 20 21:07:33.478212 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.478183 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:33.576826 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.576791 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:33.584678 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.584650 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bbdc6cd7f-c87gf" Jul 20 21:07:33.680402 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.680369 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:33.689990 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.689963 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-77c4f499cc-gvnlm"] Jul 20 21:07:33.693869 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.693841 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-77c4f499cc-gvnlm" Jul 20 21:07:33.697576 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.697541 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 20 21:07:33.697753 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.697552 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 20 21:07:33.698438 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.698415 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 20 21:07:33.698558 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.698437 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 20 21:07:33.698558 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.698498 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-85xlp\"" Jul 20 21:07:33.698558 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.698416 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-8nbhoqnemvm0u\"" Jul 20 21:07:33.723588 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.723552 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-77c4f499cc-gvnlm"] Jul 20 21:07:33.742366 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.742332 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bbdc6cd7f-c87gf"] Jul 20 21:07:33.745159 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.745115 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62ba6532-13ac-41d2-b6c4-0881a9123227-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-77c4f499cc-gvnlm\" (UID: \"62ba6532-13ac-41d2-b6c4-0881a9123227\") " pod="openshift-monitoring/metrics-server-77c4f499cc-gvnlm" Jul 20 21:07:33.745262 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.745197 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/62ba6532-13ac-41d2-b6c4-0881a9123227-client-ca-bundle\") pod \"metrics-server-77c4f499cc-gvnlm\" (UID: \"62ba6532-13ac-41d2-b6c4-0881a9123227\") " pod="openshift-monitoring/metrics-server-77c4f499cc-gvnlm" Jul 20 21:07:33.745262 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.745218 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/62ba6532-13ac-41d2-b6c4-0881a9123227-audit-log\") pod \"metrics-server-77c4f499cc-gvnlm\" (UID: \"62ba6532-13ac-41d2-b6c4-0881a9123227\") " pod="openshift-monitoring/metrics-server-77c4f499cc-gvnlm" Jul 20 21:07:33.745368 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.745305 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/62ba6532-13ac-41d2-b6c4-0881a9123227-secret-metrics-server-client-certs\") pod \"metrics-server-77c4f499cc-gvnlm\" (UID: \"62ba6532-13ac-41d2-b6c4-0881a9123227\") " pod="openshift-monitoring/metrics-server-77c4f499cc-gvnlm" Jul 20 21:07:33.745443 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.745428 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/62ba6532-13ac-41d2-b6c4-0881a9123227-secret-metrics-server-tls\") pod \"metrics-server-77c4f499cc-gvnlm\" (UID: \"62ba6532-13ac-41d2-b6c4-0881a9123227\") " pod="openshift-monitoring/metrics-server-77c4f499cc-gvnlm" Jul 20 21:07:33.745499 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.745465 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/62ba6532-13ac-41d2-b6c4-0881a9123227-metrics-server-audit-profiles\") pod \"metrics-server-77c4f499cc-gvnlm\" (UID: \"62ba6532-13ac-41d2-b6c4-0881a9123227\") " pod="openshift-monitoring/metrics-server-77c4f499cc-gvnlm" Jul 20 21:07:33.745539 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.745495 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd5dw\" (UniqueName: \"kubernetes.io/projected/62ba6532-13ac-41d2-b6c4-0881a9123227-kube-api-access-zd5dw\") pod \"metrics-server-77c4f499cc-gvnlm\" (UID: \"62ba6532-13ac-41d2-b6c4-0881a9123227\") " pod="openshift-monitoring/metrics-server-77c4f499cc-gvnlm" Jul 20 21:07:33.846342 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.846299 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/62ba6532-13ac-41d2-b6c4-0881a9123227-secret-metrics-server-tls\") pod \"metrics-server-77c4f499cc-gvnlm\" (UID: \"62ba6532-13ac-41d2-b6c4-0881a9123227\") " pod="openshift-monitoring/metrics-server-77c4f499cc-gvnlm" Jul 20 21:07:33.846531 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.846357 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/62ba6532-13ac-41d2-b6c4-0881a9123227-metrics-server-audit-profiles\") pod \"metrics-server-77c4f499cc-gvnlm\" (UID: \"62ba6532-13ac-41d2-b6c4-0881a9123227\") " pod="openshift-monitoring/metrics-server-77c4f499cc-gvnlm" Jul 20 21:07:33.846531 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.846389 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd5dw\" (UniqueName: \"kubernetes.io/projected/62ba6532-13ac-41d2-b6c4-0881a9123227-kube-api-access-zd5dw\") pod \"metrics-server-77c4f499cc-gvnlm\" (UID: \"62ba6532-13ac-41d2-b6c4-0881a9123227\") " pod="openshift-monitoring/metrics-server-77c4f499cc-gvnlm" Jul 20 21:07:33.846531 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.846420 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62ba6532-13ac-41d2-b6c4-0881a9123227-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-77c4f499cc-gvnlm\" (UID: \"62ba6532-13ac-41d2-b6c4-0881a9123227\") " pod="openshift-monitoring/metrics-server-77c4f499cc-gvnlm" Jul 20 21:07:33.846531 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.846463 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/62ba6532-13ac-41d2-b6c4-0881a9123227-client-ca-bundle\") pod \"metrics-server-77c4f499cc-gvnlm\" (UID: \"62ba6532-13ac-41d2-b6c4-0881a9123227\") " pod="openshift-monitoring/metrics-server-77c4f499cc-gvnlm" Jul 20 21:07:33.846727 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.846575 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/62ba6532-13ac-41d2-b6c4-0881a9123227-audit-log\") pod \"metrics-server-77c4f499cc-gvnlm\" (UID: \"62ba6532-13ac-41d2-b6c4-0881a9123227\") " pod="openshift-monitoring/metrics-server-77c4f499cc-gvnlm" Jul 20 21:07:33.846727 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.846614 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/62ba6532-13ac-41d2-b6c4-0881a9123227-secret-metrics-server-client-certs\") pod \"metrics-server-77c4f499cc-gvnlm\" (UID: \"62ba6532-13ac-41d2-b6c4-0881a9123227\") " pod="openshift-monitoring/metrics-server-77c4f499cc-gvnlm" Jul 20 21:07:33.847575 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.847506 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/62ba6532-13ac-41d2-b6c4-0881a9123227-audit-log\") pod \"metrics-server-77c4f499cc-gvnlm\" (UID: \"62ba6532-13ac-41d2-b6c4-0881a9123227\") " pod="openshift-monitoring/metrics-server-77c4f499cc-gvnlm" Jul 20 21:07:33.847575 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.847519 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62ba6532-13ac-41d2-b6c4-0881a9123227-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-77c4f499cc-gvnlm\" (UID: \"62ba6532-13ac-41d2-b6c4-0881a9123227\") " pod="openshift-monitoring/metrics-server-77c4f499cc-gvnlm" Jul 20 21:07:33.847804 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.847782 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/62ba6532-13ac-41d2-b6c4-0881a9123227-metrics-server-audit-profiles\") pod \"metrics-server-77c4f499cc-gvnlm\" (UID: \"62ba6532-13ac-41d2-b6c4-0881a9123227\") " pod="openshift-monitoring/metrics-server-77c4f499cc-gvnlm" Jul 20 21:07:33.849348 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.849324 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/62ba6532-13ac-41d2-b6c4-0881a9123227-secret-metrics-server-client-certs\") pod \"metrics-server-77c4f499cc-gvnlm\" (UID: \"62ba6532-13ac-41d2-b6c4-0881a9123227\") " pod="openshift-monitoring/metrics-server-77c4f499cc-gvnlm" Jul 20 21:07:33.849471 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.849444 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/62ba6532-13ac-41d2-b6c4-0881a9123227-client-ca-bundle\") pod \"metrics-server-77c4f499cc-gvnlm\" (UID: \"62ba6532-13ac-41d2-b6c4-0881a9123227\") " pod="openshift-monitoring/metrics-server-77c4f499cc-gvnlm" Jul 20 21:07:33.849536 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.849508 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/62ba6532-13ac-41d2-b6c4-0881a9123227-secret-metrics-server-tls\") pod \"metrics-server-77c4f499cc-gvnlm\" (UID: \"62ba6532-13ac-41d2-b6c4-0881a9123227\") " pod="openshift-monitoring/metrics-server-77c4f499cc-gvnlm" Jul 20 21:07:33.858682 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:33.858654 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd5dw\" (UniqueName: \"kubernetes.io/projected/62ba6532-13ac-41d2-b6c4-0881a9123227-kube-api-access-zd5dw\") pod \"metrics-server-77c4f499cc-gvnlm\" (UID: \"62ba6532-13ac-41d2-b6c4-0881a9123227\") " pod="openshift-monitoring/metrics-server-77c4f499cc-gvnlm" Jul 20 21:07:34.005950 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.005912 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-77c4f499cc-gvnlm" Jul 20 21:07:34.152081 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:07:34.151994 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b57684d_9ed6_45f7_8584_da0b459485f5.slice/crio-378ba216620133424fc826fa067ab495b224396d20a3bad74fc28914537971dd WatchSource:0}: Error finding container 378ba216620133424fc826fa067ab495b224396d20a3bad74fc28914537971dd: Status 404 returned error can't find the container with id 378ba216620133424fc826fa067ab495b224396d20a3bad74fc28914537971dd Jul 20 21:07:34.180705 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.180678 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-vccsj"] Jul 20 21:07:34.186057 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.184720 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-vccsj" Jul 20 21:07:34.188380 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.188361 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 20 21:07:34.188480 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.188398 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-6rxj2\"" Jul 20 21:07:34.193612 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.193545 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:34.199041 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.198984 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-vccsj"] Jul 20 21:07:34.210473 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.209759 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75c69c9dc9-vl49s" Jul 20 21:07:34.210473 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.209832 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-75c69c9dc9-vl49s" Jul 20 21:07:34.216204 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.216184 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-75c69c9dc9-vl49s" Jul 20 21:07:34.250557 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.250524 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4prrs\" (UniqueName: \"kubernetes.io/projected/35ab2239-07e1-4447-a1de-f3ca98dbbbc4-kube-api-access-4prrs\") pod \"monitoring-plugin-7b7b7bc8bf-vccsj\" (UID: \"35ab2239-07e1-4447-a1de-f3ca98dbbbc4\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-vccsj" Jul 20 21:07:34.250687 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.250672 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/35ab2239-07e1-4447-a1de-f3ca98dbbbc4-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-vccsj\" (UID: \"35ab2239-07e1-4447-a1de-f3ca98dbbbc4\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-vccsj" Jul 20 21:07:34.284932 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.284891 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:34.301380 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.301356 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-77c4f499cc-gvnlm"] Jul 20 21:07:34.305047 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:07:34.305020 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62ba6532_13ac_41d2_b6c4_0881a9123227.slice/crio-87b18d333b5bd0841874b9c474c286aa544428564e67159533ec8db6c88c8e68 WatchSource:0}: Error finding container 87b18d333b5bd0841874b9c474c286aa544428564e67159533ec8db6c88c8e68: Status 404 returned error can't find the container with id 87b18d333b5bd0841874b9c474c286aa544428564e67159533ec8db6c88c8e68 Jul 20 21:07:34.351843 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.351817 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/35ab2239-07e1-4447-a1de-f3ca98dbbbc4-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-vccsj\" (UID: \"35ab2239-07e1-4447-a1de-f3ca98dbbbc4\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-vccsj" Jul 20 21:07:34.351985 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.351879 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4prrs\" (UniqueName: \"kubernetes.io/projected/35ab2239-07e1-4447-a1de-f3ca98dbbbc4-kube-api-access-4prrs\") pod \"monitoring-plugin-7b7b7bc8bf-vccsj\" (UID: \"35ab2239-07e1-4447-a1de-f3ca98dbbbc4\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-vccsj" Jul 20 21:07:34.354317 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.354295 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/35ab2239-07e1-4447-a1de-f3ca98dbbbc4-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-vccsj\" (UID: \"35ab2239-07e1-4447-a1de-f3ca98dbbbc4\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-vccsj" Jul 20 21:07:34.360530 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.360503 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4prrs\" (UniqueName: \"kubernetes.io/projected/35ab2239-07e1-4447-a1de-f3ca98dbbbc4-kube-api-access-4prrs\") pod \"monitoring-plugin-7b7b7bc8bf-vccsj\" (UID: \"35ab2239-07e1-4447-a1de-f3ca98dbbbc4\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-vccsj" Jul 20 21:07:34.385644 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.385606 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:34.497593 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.497564 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-vccsj" Jul 20 21:07:34.586496 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.586463 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-84ddb46cc7-r6khm"] Jul 20 21:07:34.595817 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.595785 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-84ddb46cc7-r6khm" Jul 20 21:07:34.601921 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.601892 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 20 21:07:34.602084 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.602042 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 20 21:07:34.602084 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.602050 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 20 21:07:34.602417 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.602095 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 20 21:07:34.602417 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.602220 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 20 21:07:34.603012 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.602994 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-jjxpf\"" Jul 20 21:07:34.610512 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.610491 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 20 21:07:34.613991 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.613969 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-84ddb46cc7-r6khm"] Jul 20 21:07:34.636845 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.636820 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-vccsj"] Jul 20 21:07:34.640338 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:07:34.640314 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35ab2239_07e1_4447_a1de_f3ca98dbbbc4.slice/crio-af0f455878e963585588b8a0b993f3e1ba947e610ad5ddb62b9560c1092c764a WatchSource:0}: Error finding container af0f455878e963585588b8a0b993f3e1ba947e610ad5ddb62b9560c1092c764a: Status 404 returned error can't find the container with id af0f455878e963585588b8a0b993f3e1ba947e610ad5ddb62b9560c1092c764a Jul 20 21:07:34.653864 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.653835 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwz49\" (UniqueName: \"kubernetes.io/projected/a16c41e7-0bed-413b-abbb-17ef8ba56057-kube-api-access-bwz49\") pod \"telemeter-client-84ddb46cc7-r6khm\" (UID: \"a16c41e7-0bed-413b-abbb-17ef8ba56057\") " pod="openshift-monitoring/telemeter-client-84ddb46cc7-r6khm" Jul 20 21:07:34.653959 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.653885 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a16c41e7-0bed-413b-abbb-17ef8ba56057-metrics-client-ca\") pod \"telemeter-client-84ddb46cc7-r6khm\" (UID: \"a16c41e7-0bed-413b-abbb-17ef8ba56057\") " pod="openshift-monitoring/telemeter-client-84ddb46cc7-r6khm" Jul 20 21:07:34.654003 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.653954 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a16c41e7-0bed-413b-abbb-17ef8ba56057-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-84ddb46cc7-r6khm\" (UID: \"a16c41e7-0bed-413b-abbb-17ef8ba56057\") " pod="openshift-monitoring/telemeter-client-84ddb46cc7-r6khm" Jul 20 21:07:34.654003 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.653989 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a16c41e7-0bed-413b-abbb-17ef8ba56057-telemeter-trusted-ca-bundle\") pod \"telemeter-client-84ddb46cc7-r6khm\" (UID: \"a16c41e7-0bed-413b-abbb-17ef8ba56057\") " pod="openshift-monitoring/telemeter-client-84ddb46cc7-r6khm" Jul 20 21:07:34.654099 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.654036 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a16c41e7-0bed-413b-abbb-17ef8ba56057-secret-telemeter-client\") pod \"telemeter-client-84ddb46cc7-r6khm\" (UID: \"a16c41e7-0bed-413b-abbb-17ef8ba56057\") " pod="openshift-monitoring/telemeter-client-84ddb46cc7-r6khm" Jul 20 21:07:34.654099 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.654051 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a16c41e7-0bed-413b-abbb-17ef8ba56057-telemeter-client-tls\") pod \"telemeter-client-84ddb46cc7-r6khm\" (UID: \"a16c41e7-0bed-413b-abbb-17ef8ba56057\") " pod="openshift-monitoring/telemeter-client-84ddb46cc7-r6khm" Jul 20 21:07:34.654099 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.654079 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a16c41e7-0bed-413b-abbb-17ef8ba56057-federate-client-tls\") pod \"telemeter-client-84ddb46cc7-r6khm\" (UID: \"a16c41e7-0bed-413b-abbb-17ef8ba56057\") " pod="openshift-monitoring/telemeter-client-84ddb46cc7-r6khm" Jul 20 21:07:34.654245 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.654122 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a16c41e7-0bed-413b-abbb-17ef8ba56057-serving-certs-ca-bundle\") pod \"telemeter-client-84ddb46cc7-r6khm\" (UID: \"a16c41e7-0bed-413b-abbb-17ef8ba56057\") " pod="openshift-monitoring/telemeter-client-84ddb46cc7-r6khm" Jul 20 21:07:34.755543 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.755453 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwz49\" (UniqueName: \"kubernetes.io/projected/a16c41e7-0bed-413b-abbb-17ef8ba56057-kube-api-access-bwz49\") pod \"telemeter-client-84ddb46cc7-r6khm\" (UID: \"a16c41e7-0bed-413b-abbb-17ef8ba56057\") " pod="openshift-monitoring/telemeter-client-84ddb46cc7-r6khm" Jul 20 21:07:34.755543 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.755510 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a16c41e7-0bed-413b-abbb-17ef8ba56057-metrics-client-ca\") pod \"telemeter-client-84ddb46cc7-r6khm\" (UID: \"a16c41e7-0bed-413b-abbb-17ef8ba56057\") " pod="openshift-monitoring/telemeter-client-84ddb46cc7-r6khm" Jul 20 21:07:34.755543 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.755544 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a16c41e7-0bed-413b-abbb-17ef8ba56057-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-84ddb46cc7-r6khm\" (UID: \"a16c41e7-0bed-413b-abbb-17ef8ba56057\") " pod="openshift-monitoring/telemeter-client-84ddb46cc7-r6khm" Jul 20 21:07:34.755810 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.755567 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a16c41e7-0bed-413b-abbb-17ef8ba56057-telemeter-trusted-ca-bundle\") pod \"telemeter-client-84ddb46cc7-r6khm\" (UID: \"a16c41e7-0bed-413b-abbb-17ef8ba56057\") " pod="openshift-monitoring/telemeter-client-84ddb46cc7-r6khm" Jul 20 21:07:34.755810 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.755600 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a16c41e7-0bed-413b-abbb-17ef8ba56057-secret-telemeter-client\") pod \"telemeter-client-84ddb46cc7-r6khm\" (UID: \"a16c41e7-0bed-413b-abbb-17ef8ba56057\") " pod="openshift-monitoring/telemeter-client-84ddb46cc7-r6khm" Jul 20 21:07:34.755810 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.755661 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a16c41e7-0bed-413b-abbb-17ef8ba56057-telemeter-client-tls\") pod \"telemeter-client-84ddb46cc7-r6khm\" (UID: \"a16c41e7-0bed-413b-abbb-17ef8ba56057\") " pod="openshift-monitoring/telemeter-client-84ddb46cc7-r6khm" Jul 20 21:07:34.755810 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.755696 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a16c41e7-0bed-413b-abbb-17ef8ba56057-federate-client-tls\") pod \"telemeter-client-84ddb46cc7-r6khm\" (UID: \"a16c41e7-0bed-413b-abbb-17ef8ba56057\") " pod="openshift-monitoring/telemeter-client-84ddb46cc7-r6khm" Jul 20 21:07:34.755810 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.755717 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a16c41e7-0bed-413b-abbb-17ef8ba56057-serving-certs-ca-bundle\") pod \"telemeter-client-84ddb46cc7-r6khm\" (UID: \"a16c41e7-0bed-413b-abbb-17ef8ba56057\") " pod="openshift-monitoring/telemeter-client-84ddb46cc7-r6khm" Jul 20 21:07:34.756498 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.756436 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a16c41e7-0bed-413b-abbb-17ef8ba56057-serving-certs-ca-bundle\") pod \"telemeter-client-84ddb46cc7-r6khm\" (UID: \"a16c41e7-0bed-413b-abbb-17ef8ba56057\") " pod="openshift-monitoring/telemeter-client-84ddb46cc7-r6khm" Jul 20 21:07:34.756616 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.756519 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a16c41e7-0bed-413b-abbb-17ef8ba56057-telemeter-trusted-ca-bundle\") pod \"telemeter-client-84ddb46cc7-r6khm\" (UID: \"a16c41e7-0bed-413b-abbb-17ef8ba56057\") " pod="openshift-monitoring/telemeter-client-84ddb46cc7-r6khm" Jul 20 21:07:34.756616 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.756547 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a16c41e7-0bed-413b-abbb-17ef8ba56057-metrics-client-ca\") pod \"telemeter-client-84ddb46cc7-r6khm\" (UID: \"a16c41e7-0bed-413b-abbb-17ef8ba56057\") " pod="openshift-monitoring/telemeter-client-84ddb46cc7-r6khm" Jul 20 21:07:34.758176 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.758150 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a16c41e7-0bed-413b-abbb-17ef8ba56057-federate-client-tls\") pod \"telemeter-client-84ddb46cc7-r6khm\" (UID: \"a16c41e7-0bed-413b-abbb-17ef8ba56057\") " pod="openshift-monitoring/telemeter-client-84ddb46cc7-r6khm" Jul 20 21:07:34.758264 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.758206 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a16c41e7-0bed-413b-abbb-17ef8ba56057-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-84ddb46cc7-r6khm\" (UID: \"a16c41e7-0bed-413b-abbb-17ef8ba56057\") " pod="openshift-monitoring/telemeter-client-84ddb46cc7-r6khm" Jul 20 21:07:34.758264 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.758237 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a16c41e7-0bed-413b-abbb-17ef8ba56057-telemeter-client-tls\") pod \"telemeter-client-84ddb46cc7-r6khm\" (UID: \"a16c41e7-0bed-413b-abbb-17ef8ba56057\") " pod="openshift-monitoring/telemeter-client-84ddb46cc7-r6khm" Jul 20 21:07:34.758378 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.758360 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a16c41e7-0bed-413b-abbb-17ef8ba56057-secret-telemeter-client\") pod \"telemeter-client-84ddb46cc7-r6khm\" (UID: \"a16c41e7-0bed-413b-abbb-17ef8ba56057\") " pod="openshift-monitoring/telemeter-client-84ddb46cc7-r6khm" Jul 20 21:07:34.764497 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.764474 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwz49\" (UniqueName: \"kubernetes.io/projected/a16c41e7-0bed-413b-abbb-17ef8ba56057-kube-api-access-bwz49\") pod \"telemeter-client-84ddb46cc7-r6khm\" (UID: \"a16c41e7-0bed-413b-abbb-17ef8ba56057\") " pod="openshift-monitoring/telemeter-client-84ddb46cc7-r6khm" Jul 20 21:07:34.908198 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:34.908158 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-84ddb46cc7-r6khm" Jul 20 21:07:35.068193 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.068162 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-84ddb46cc7-r6khm"] Jul 20 21:07:35.070917 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:07:35.070888 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda16c41e7_0bed_413b_abbb_17ef8ba56057.slice/crio-ee445e8222e52c1cae1fde9ebbdd72c464b1cadd52d9b69c63d1ff5ef8318ba9 WatchSource:0}: Error finding container ee445e8222e52c1cae1fde9ebbdd72c464b1cadd52d9b69c63d1ff5ef8318ba9: Status 404 returned error can't find the container with id ee445e8222e52c1cae1fde9ebbdd72c464b1cadd52d9b69c63d1ff5ef8318ba9 Jul 20 21:07:35.135532 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.135494 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-84ddb46cc7-r6khm" event={"ID":"a16c41e7-0bed-413b-abbb-17ef8ba56057","Type":"ContainerStarted","Data":"ee445e8222e52c1cae1fde9ebbdd72c464b1cadd52d9b69c63d1ff5ef8318ba9"} Jul 20 21:07:35.137884 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.137854 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c8d56f8b8-hxvgr" event={"ID":"ff6806f9-668b-4512-906a-df4a69b73ba9","Type":"ContainerStarted","Data":"cc5d2e965fe28290d1bc63f734b1d08fa9cefd4d1eddb38538db5a0398c53217"} Jul 20 21:07:35.138002 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.137917 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c8d56f8b8-hxvgr" event={"ID":"ff6806f9-668b-4512-906a-df4a69b73ba9","Type":"ContainerStarted","Data":"6bbd9f07ba36d918cf66fe9dbd8098e33febd926475e8b1d561f6c09af2690a5"} Jul 20 21:07:35.138002 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.137957 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c8d56f8b8-hxvgr" event={"ID":"ff6806f9-668b-4512-906a-df4a69b73ba9","Type":"ContainerStarted","Data":"3cd191e3014b8a23e039be1b0fb14d95bce44da5404a90f5d88e5ad5902ebb81"} Jul 20 21:07:35.139145 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.139099 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-vccsj" event={"ID":"35ab2239-07e1-4447-a1de-f3ca98dbbbc4","Type":"ContainerStarted","Data":"af0f455878e963585588b8a0b993f3e1ba947e610ad5ddb62b9560c1092c764a"} Jul 20 21:07:35.140532 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.140503 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-77c4f499cc-gvnlm" event={"ID":"62ba6532-13ac-41d2-b6c4-0881a9123227","Type":"ContainerStarted","Data":"87b18d333b5bd0841874b9c474c286aa544428564e67159533ec8db6c88c8e68"} Jul 20 21:07:35.142100 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.142071 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bbdc6cd7f-c87gf" event={"ID":"4b57684d-9ed6-45f7-8584-da0b459485f5","Type":"ContainerStarted","Data":"89507cc5e5a57c514fa0d983377d074e329d73b78a33a2dbcc5025167d4776ff"} Jul 20 21:07:35.142233 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.142118 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bbdc6cd7f-c87gf" event={"ID":"4b57684d-9ed6-45f7-8584-da0b459485f5","Type":"ContainerStarted","Data":"378ba216620133424fc826fa067ab495b224396d20a3bad74fc28914537971dd"} Jul 20 21:07:35.146747 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.146720 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-75c69c9dc9-vl49s" Jul 20 21:07:35.162410 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.162360 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5bbdc6cd7f-c87gf" podStartSLOduration=2.16234658 podStartE2EDuration="2.16234658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 21:07:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:07:35.161841645 +0000 UTC m=+294.487544589" watchObservedRunningTime="2026-07-20 21:07:35.16234658 +0000 UTC m=+294.488049544" Jul 20 21:07:35.609649 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.609563 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:35.609649 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.609633 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 20 21:07:35.614052 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.614027 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.617447 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.617037 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 20 21:07:35.617447 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.617085 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9u1e07mk6t0nk\"" Jul 20 21:07:35.617447 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.617391 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 20 21:07:35.623649 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.623489 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 20 21:07:35.623929 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.623842 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 20 21:07:35.624332 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.624155 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 20 21:07:35.624332 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.624200 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 20 21:07:35.624332 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.624162 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 20 21:07:35.624332 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.624267 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 20 21:07:35.624827 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.624609 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-65242\"" Jul 20 21:07:35.624827 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.624640 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 20 21:07:35.625281 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.625173 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 20 21:07:35.630958 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.630182 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 20 21:07:35.630958 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.630780 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 20 21:07:35.636662 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.636634 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 20 21:07:35.638521 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.638494 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 20 21:07:35.649622 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.649484 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 20 21:07:35.665913 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.664401 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.665913 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.664453 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.665913 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.664482 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.665913 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.664517 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.665913 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.664553 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.665913 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.664588 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-config\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.665913 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.664626 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.665913 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.664687 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-config-out\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.665913 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.664710 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.665913 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.664739 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.665913 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.664768 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.665913 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.664796 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.665913 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.664819 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.666658 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.664843 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.666658 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.664878 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.666658 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.664905 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.666658 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.664927 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.666658 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.664950 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz56g\" (UniqueName: \"kubernetes.io/projected/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-kube-api-access-sz56g\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.666658 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.664980 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.666658 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.665043 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-web-config\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.766203 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.766157 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-config\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.766384 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.766220 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.766384 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.766281 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-config-out\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.766384 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.766306 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.766384 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.766334 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.766384 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.766364 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.766655 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.766389 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.766655 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.766414 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.766655 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.766441 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.766655 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.766476 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.766655 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.766502 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.766655 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.766526 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.766655 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.766550 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz56g\" (UniqueName: \"kubernetes.io/projected/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-kube-api-access-sz56g\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.766655 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.766581 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.766655 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.766637 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-web-config\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.767072 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.766688 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.767072 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.766712 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.767072 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.766737 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.767072 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.766767 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.767072 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.766807 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.768094 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.768051 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.769356 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.768827 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.769356 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.769001 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.769498 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.769462 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.769498 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.769470 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.769608 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.769543 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.770666 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.770105 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-config\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.770666 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.770379 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.772275 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.772246 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.773101 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.773059 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.773489 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.773252 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-config-out\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.773712 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.773693 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-web-config\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.774390 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.773975 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.774390 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.774341 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.775521 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.775491 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.776718 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.776650 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.776718 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.776677 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.776946 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.776905 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.777759 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.777704 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.778348 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.778328 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz56g\" (UniqueName: \"kubernetes.io/projected/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-kube-api-access-sz56g\") pod \"prometheus-k8s-0\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:35.933030 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:35.932947 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:36.695959 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:36.695936 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 20 21:07:37.149031 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:07:37.149004 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4dd4ce8_4f27_44f8_b790_66afc1d0e273.slice/crio-287003b5561d14d1a620354f3f268e6d860a83d1fb34843b56ade69c81988387 WatchSource:0}: Error finding container 287003b5561d14d1a620354f3f268e6d860a83d1fb34843b56ade69c81988387: Status 404 returned error can't find the container with id 287003b5561d14d1a620354f3f268e6d860a83d1fb34843b56ade69c81988387 Jul 20 21:07:37.154536 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:37.154506 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c8d56f8b8-hxvgr" event={"ID":"ff6806f9-668b-4512-906a-df4a69b73ba9","Type":"ContainerStarted","Data":"fe2b6f81312271d4762db99a8085dc83bfd5dcd78d26057a31b33c6123055066"} Jul 20 21:07:37.154605 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:37.154589 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c8d56f8b8-hxvgr" event={"ID":"ff6806f9-668b-4512-906a-df4a69b73ba9","Type":"ContainerStarted","Data":"0d2dd86f6e3c2b61d51ba4949552b55d98451136737903fa275d82a7e52ce44f"} Jul 20 21:07:37.157088 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:37.157057 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-vccsj" event={"ID":"35ab2239-07e1-4447-a1de-f3ca98dbbbc4","Type":"ContainerStarted","Data":"f642f7c4331d0cfaffa26eb01efb4f668bee399009f8a9585c4bf3a1ffdf2464"} Jul 20 21:07:37.157205 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:37.157119 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-vccsj" Jul 20 21:07:37.159287 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:37.159265 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-77c4f499cc-gvnlm" event={"ID":"62ba6532-13ac-41d2-b6c4-0881a9123227","Type":"ContainerStarted","Data":"a2c087ad3b69c67d6b195b0d0a433660e498590b2899fe0aef84c94b0afb2176"} Jul 20 21:07:37.162279 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:37.162260 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-vccsj" Jul 20 21:07:37.175932 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:37.175891 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-vccsj" podStartSLOduration=1.300240146 podStartE2EDuration="3.17587587s" totalImagesPullingTime="1.875635724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 21:07:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:07:37.173965376 +0000 UTC m=+296.499668320" watchObservedRunningTime="2026-07-20 21:07:37.17587587 +0000 UTC m=+296.501578810" Jul 20 21:07:37.193078 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:37.193034 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-77c4f499cc-gvnlm" podStartSLOduration=1.987299162 podStartE2EDuration="4.19301897s" totalImagesPullingTime="2.205719808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 21:07:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:07:37.191731668 +0000 UTC m=+296.517434614" watchObservedRunningTime="2026-07-20 21:07:37.19301897 +0000 UTC m=+296.518721912" Jul 20 21:07:38.130977 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:38.130938 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:38.165708 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:38.165669 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-84ddb46cc7-r6khm" event={"ID":"a16c41e7-0bed-413b-abbb-17ef8ba56057","Type":"ContainerStarted","Data":"92120e4266c1305b0307344f08f535a0468ebd9fd3e99adb4d98984ba18d46a7"} Jul 20 21:07:38.169171 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:38.169118 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c8d56f8b8-hxvgr" event={"ID":"ff6806f9-668b-4512-906a-df4a69b73ba9","Type":"ContainerStarted","Data":"c1a2c76ca341c944e87f83d6b3e9714a7995a01fcb5ceca58b416e89be5d785c"} Jul 20 21:07:38.169313 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:38.169275 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5c8d56f8b8-hxvgr" Jul 20 21:07:38.170581 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:38.170552 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4dd4ce8-4f27-44f8-b790-66afc1d0e273","Type":"ContainerStarted","Data":"287003b5561d14d1a620354f3f268e6d860a83d1fb34843b56ade69c81988387"} Jul 20 21:07:38.194942 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:38.194765 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5c8d56f8b8-hxvgr" podStartSLOduration=2.701127357 podStartE2EDuration="7.194743358s" totalImagesPullingTime="4.493616001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 21:07:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:07:38.192305044 +0000 UTC m=+297.518007994" watchObservedRunningTime="2026-07-20 21:07:38.194743358 +0000 UTC m=+297.520446302" Jul 20 21:07:38.234628 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:38.234596 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:38.265938 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:07:38.265899 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-t79t8" podUID="0f09f6b9-6ed0-4165-bc40-706eb42f8f0d" Jul 20 21:07:39.177962 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:39.177925 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-84ddb46cc7-r6khm" event={"ID":"a16c41e7-0bed-413b-abbb-17ef8ba56057","Type":"ContainerStarted","Data":"8577bc5f591b68f482a706c26b7ab9672bfd11fd94c1dc54fcfc467686af58f1"} Jul 20 21:07:39.178465 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:39.177987 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-84ddb46cc7-r6khm" event={"ID":"a16c41e7-0bed-413b-abbb-17ef8ba56057","Type":"ContainerStarted","Data":"6b98a9344ed90b9c437aba61b7aadce5080a8f803e0345963ceb1de0cad80249"} Jul 20 21:07:39.179505 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:39.179480 2562 generic.go:356] "Generic (PLEG): container finished" podID="b4dd4ce8-4f27-44f8-b790-66afc1d0e273" containerID="144911b5b9578f85d42ea90f280fa17c3af03ea7ca5a916fb0e043e508502988" exitCode=0 Jul 20 21:07:39.179630 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:39.179577 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4dd4ce8-4f27-44f8-b790-66afc1d0e273","Type":"ContainerDied","Data":"144911b5b9578f85d42ea90f280fa17c3af03ea7ca5a916fb0e043e508502988"} Jul 20 21:07:39.185914 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:39.185894 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5c8d56f8b8-hxvgr" Jul 20 21:07:39.202044 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:39.202006 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-84ddb46cc7-r6khm" podStartSLOduration=1.99033671 podStartE2EDuration="5.201993663s" totalImagesPullingTime="3.211656953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 21:07:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:07:39.20169552 +0000 UTC m=+298.527398462" watchObservedRunningTime="2026-07-20 21:07:39.201993663 +0000 UTC m=+298.527696605" Jul 20 21:07:40.110448 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:40.110418 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zp575" Jul 20 21:07:40.724233 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:40.724195 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:40.824732 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:40.824697 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:41.737569 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:41.737532 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:41.837836 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:41.837792 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:41.926260 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:41.926221 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bbdc6cd7f-c87gf"] Jul 20 21:07:42.205363 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:42.205336 2562 kubelet.go:1648] "Image garbage collection succeeded" Jul 20 21:07:42.791716 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:42.791682 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:42.854635 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:42.854599 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:42.925406 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:42.925371 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:43.025476 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:43.025443 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:43.199751 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:43.199663 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4dd4ce8-4f27-44f8-b790-66afc1d0e273","Type":"ContainerStarted","Data":"2fb8053d399a787aebf23a8ab2dfc2caaee4c0a60d3439e926da4094ad35e59c"} Jul 20 21:07:43.200090 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:43.199754 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4dd4ce8-4f27-44f8-b790-66afc1d0e273","Type":"ContainerStarted","Data":"a55527356515053396018dc3293569fd7f37746763196bfa65f3f5d294686785"} Jul 20 21:07:43.200090 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:43.199781 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4dd4ce8-4f27-44f8-b790-66afc1d0e273","Type":"ContainerStarted","Data":"5084e80170cb16afece338654b6fc838047c3e7fc692e4a69bc60fa2b4a9c54d"} Jul 20 21:07:43.200090 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:43.199807 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4dd4ce8-4f27-44f8-b790-66afc1d0e273","Type":"ContainerStarted","Data":"3d174c39ac865c56b23aa9724d3d0ab339670a8db7d95a4b90a14eeec2e662e5"} Jul 20 21:07:43.200090 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:43.199831 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4dd4ce8-4f27-44f8-b790-66afc1d0e273","Type":"ContainerStarted","Data":"e6aad72ba73cd8452e8fd0e4182ac4b8a82f41d5367d01d7a1d1a5295237079c"} Jul 20 21:07:43.200090 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:43.199857 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4dd4ce8-4f27-44f8-b790-66afc1d0e273","Type":"ContainerStarted","Data":"3ea33f05351732d65f49304f67a74070a95c3e524e3892b1b197f1213fb12a8f"} Jul 20 21:07:43.230151 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:43.230017 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.95555906 podStartE2EDuration="8.229996392s" totalImagesPullingTime="4.248073986s" totalInitContainerRuntime="26.363346ms" isStatefulPod=false podCreationTimestamp="2026-07-20 21:07:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:07:43.228526875 +0000 UTC m=+302.554229846" watchObservedRunningTime="2026-07-20 21:07:43.229996392 +0000 UTC m=+302.555699334" Jul 20 21:07:43.585793 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:43.585756 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5bbdc6cd7f-c87gf" Jul 20 21:07:45.117750 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:45.117685 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5748dd95bb-qhfpf" podUID="8e63b84e-6cf9-467e-b114-9682be23c2b9" containerName="registry" containerID="cri-o://6d706271b8fa258ec214c6bf40bb0a4a51a81f1cbcce92f7195a429afc34cb00" gracePeriod=30 Jul 20 21:07:45.368246 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:45.368188 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5748dd95bb-qhfpf" Jul 20 21:07:45.465561 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:45.465527 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8e63b84e-6cf9-467e-b114-9682be23c2b9-installation-pull-secrets\") pod \"8e63b84e-6cf9-467e-b114-9682be23c2b9\" (UID: \"8e63b84e-6cf9-467e-b114-9682be23c2b9\") " Jul 20 21:07:45.465561 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:45.465565 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8e63b84e-6cf9-467e-b114-9682be23c2b9-ca-trust-extracted\") pod \"8e63b84e-6cf9-467e-b114-9682be23c2b9\" (UID: \"8e63b84e-6cf9-467e-b114-9682be23c2b9\") " Jul 20 21:07:45.465764 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:45.465650 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8e63b84e-6cf9-467e-b114-9682be23c2b9-trusted-ca\") pod \"8e63b84e-6cf9-467e-b114-9682be23c2b9\" (UID: \"8e63b84e-6cf9-467e-b114-9682be23c2b9\") " Jul 20 21:07:45.465764 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:45.465683 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8e63b84e-6cf9-467e-b114-9682be23c2b9-registry-tls\") pod \"8e63b84e-6cf9-467e-b114-9682be23c2b9\" (UID: \"8e63b84e-6cf9-467e-b114-9682be23c2b9\") " Jul 20 21:07:45.465764 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:45.465717 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8e63b84e-6cf9-467e-b114-9682be23c2b9-image-registry-private-configuration\") pod \"8e63b84e-6cf9-467e-b114-9682be23c2b9\" (UID: \"8e63b84e-6cf9-467e-b114-9682be23c2b9\") " Jul 20 21:07:45.465764 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:45.465760 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mfxjh\" (UniqueName: \"kubernetes.io/projected/8e63b84e-6cf9-467e-b114-9682be23c2b9-kube-api-access-mfxjh\") pod \"8e63b84e-6cf9-467e-b114-9682be23c2b9\" (UID: \"8e63b84e-6cf9-467e-b114-9682be23c2b9\") " Jul 20 21:07:45.465957 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:45.465906 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8e63b84e-6cf9-467e-b114-9682be23c2b9-registry-certificates\") pod \"8e63b84e-6cf9-467e-b114-9682be23c2b9\" (UID: \"8e63b84e-6cf9-467e-b114-9682be23c2b9\") " Jul 20 21:07:45.466013 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:45.465972 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8e63b84e-6cf9-467e-b114-9682be23c2b9-bound-sa-token\") pod \"8e63b84e-6cf9-467e-b114-9682be23c2b9\" (UID: \"8e63b84e-6cf9-467e-b114-9682be23c2b9\") " Jul 20 21:07:45.466387 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:45.466217 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e63b84e-6cf9-467e-b114-9682be23c2b9-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "8e63b84e-6cf9-467e-b114-9682be23c2b9" (UID: "8e63b84e-6cf9-467e-b114-9682be23c2b9"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 21:07:45.466387 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:45.466295 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e63b84e-6cf9-467e-b114-9682be23c2b9-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "8e63b84e-6cf9-467e-b114-9682be23c2b9" (UID: "8e63b84e-6cf9-467e-b114-9682be23c2b9"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 21:07:45.466387 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:45.466360 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8e63b84e-6cf9-467e-b114-9682be23c2b9-trusted-ca\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:07:45.468040 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:45.468010 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e63b84e-6cf9-467e-b114-9682be23c2b9-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "8e63b84e-6cf9-467e-b114-9682be23c2b9" (UID: "8e63b84e-6cf9-467e-b114-9682be23c2b9"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 21:07:45.468229 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:45.468082 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e63b84e-6cf9-467e-b114-9682be23c2b9-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "8e63b84e-6cf9-467e-b114-9682be23c2b9" (UID: "8e63b84e-6cf9-467e-b114-9682be23c2b9"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 21:07:45.468376 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:45.468356 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e63b84e-6cf9-467e-b114-9682be23c2b9-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "8e63b84e-6cf9-467e-b114-9682be23c2b9" (UID: "8e63b84e-6cf9-467e-b114-9682be23c2b9"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 21:07:45.468505 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:45.468488 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e63b84e-6cf9-467e-b114-9682be23c2b9-kube-api-access-mfxjh" (OuterVolumeSpecName: "kube-api-access-mfxjh") pod "8e63b84e-6cf9-467e-b114-9682be23c2b9" (UID: "8e63b84e-6cf9-467e-b114-9682be23c2b9"). InnerVolumeSpecName "kube-api-access-mfxjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 21:07:45.468551 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:45.468511 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e63b84e-6cf9-467e-b114-9682be23c2b9-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "8e63b84e-6cf9-467e-b114-9682be23c2b9" (UID: "8e63b84e-6cf9-467e-b114-9682be23c2b9"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 21:07:45.474519 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:45.474493 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e63b84e-6cf9-467e-b114-9682be23c2b9-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "8e63b84e-6cf9-467e-b114-9682be23c2b9" (UID: "8e63b84e-6cf9-467e-b114-9682be23c2b9"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 21:07:45.567684 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:45.567631 2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8e63b84e-6cf9-467e-b114-9682be23c2b9-registry-tls\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:07:45.567684 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:45.567675 2562 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8e63b84e-6cf9-467e-b114-9682be23c2b9-image-registry-private-configuration\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:07:45.567684 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:45.567686 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfxjh\" (UniqueName: \"kubernetes.io/projected/8e63b84e-6cf9-467e-b114-9682be23c2b9-kube-api-access-mfxjh\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:07:45.567684 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:45.567697 2562 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8e63b84e-6cf9-467e-b114-9682be23c2b9-registry-certificates\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:07:45.567684 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:45.567706 2562 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8e63b84e-6cf9-467e-b114-9682be23c2b9-bound-sa-token\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:07:45.567982 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:45.567715 2562 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8e63b84e-6cf9-467e-b114-9682be23c2b9-installation-pull-secrets\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:07:45.567982 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:45.567725 2562 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8e63b84e-6cf9-467e-b114-9682be23c2b9-ca-trust-extracted\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:07:45.934152 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:45.934087 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:07:46.215025 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:46.214933 2562 generic.go:356] "Generic (PLEG): container finished" podID="8e63b84e-6cf9-467e-b114-9682be23c2b9" containerID="6d706271b8fa258ec214c6bf40bb0a4a51a81f1cbcce92f7195a429afc34cb00" exitCode=0 Jul 20 21:07:46.215025 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:46.214981 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5748dd95bb-qhfpf" Jul 20 21:07:46.215593 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:46.214983 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5748dd95bb-qhfpf" event={"ID":"8e63b84e-6cf9-467e-b114-9682be23c2b9","Type":"ContainerDied","Data":"6d706271b8fa258ec214c6bf40bb0a4a51a81f1cbcce92f7195a429afc34cb00"} Jul 20 21:07:46.215593 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:46.215098 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5748dd95bb-qhfpf" event={"ID":"8e63b84e-6cf9-467e-b114-9682be23c2b9","Type":"ContainerDied","Data":"79c23fa2896e9f2766d62d028bb2e47435dfcd4c92c2fefb679017c23e8a7467"} Jul 20 21:07:46.215593 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:46.215130 2562 scope.go:117] "RemoveContainer" containerID="6d706271b8fa258ec214c6bf40bb0a4a51a81f1cbcce92f7195a429afc34cb00" Jul 20 21:07:46.224242 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:46.224215 2562 scope.go:117] "RemoveContainer" containerID="6d706271b8fa258ec214c6bf40bb0a4a51a81f1cbcce92f7195a429afc34cb00" Jul 20 21:07:46.224560 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:07:46.224536 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d706271b8fa258ec214c6bf40bb0a4a51a81f1cbcce92f7195a429afc34cb00\": container with ID starting with 6d706271b8fa258ec214c6bf40bb0a4a51a81f1cbcce92f7195a429afc34cb00 not found: ID does not exist" containerID="6d706271b8fa258ec214c6bf40bb0a4a51a81f1cbcce92f7195a429afc34cb00" Jul 20 21:07:46.224653 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:46.224575 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d706271b8fa258ec214c6bf40bb0a4a51a81f1cbcce92f7195a429afc34cb00"} err="failed to get container status \"6d706271b8fa258ec214c6bf40bb0a4a51a81f1cbcce92f7195a429afc34cb00\": rpc error: code = NotFound desc = could not find container \"6d706271b8fa258ec214c6bf40bb0a4a51a81f1cbcce92f7195a429afc34cb00\": container with ID starting with 6d706271b8fa258ec214c6bf40bb0a4a51a81f1cbcce92f7195a429afc34cb00 not found: ID does not exist" Jul 20 21:07:46.238448 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:46.238414 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5748dd95bb-qhfpf"] Jul 20 21:07:46.243117 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:46.243087 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5748dd95bb-qhfpf"] Jul 20 21:07:46.277057 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:46.277020 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:46.331781 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:46.331740 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:46.821700 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:46.821670 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:46.925675 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:46.925642 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:47.117699 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:47.117592 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:47.177910 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:47.177881 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:47.275855 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:47.275815 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e63b84e-6cf9-467e-b114-9682be23c2b9" path="/var/lib/kubelet/pods/8e63b84e-6cf9-467e-b114-9682be23c2b9/volumes" Jul 20 21:07:52.876205 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:52.876169 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75c69c9dc9-vl49s"] Jul 20 21:07:52.977582 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:52.977537 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:53.034641 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:53.034600 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:53.095276 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:53.095242 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:53.264346 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:53.264316 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t79t8" Jul 20 21:07:54.006457 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:54.006425 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-77c4f499cc-gvnlm" Jul 20 21:07:54.006843 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:54.006482 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-77c4f499cc-gvnlm" Jul 20 21:07:55.824529 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:55.824495 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:07:55.922089 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:07:55.922045 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:08:06.951619 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:06.951558 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5bbdc6cd7f-c87gf" podUID="4b57684d-9ed6-45f7-8584-da0b459485f5" containerName="console" containerID="cri-o://89507cc5e5a57c514fa0d983377d074e329d73b78a33a2dbcc5025167d4776ff" gracePeriod=15 Jul 20 21:08:07.204900 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:07.204832 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bbdc6cd7f-c87gf" Jul 20 21:08:07.262953 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:07.262912 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4b57684d-9ed6-45f7-8584-da0b459485f5-service-ca\") pod \"4b57684d-9ed6-45f7-8584-da0b459485f5\" (UID: \"4b57684d-9ed6-45f7-8584-da0b459485f5\") " Jul 20 21:08:07.263152 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:07.262995 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4b57684d-9ed6-45f7-8584-da0b459485f5-console-oauth-config\") pod \"4b57684d-9ed6-45f7-8584-da0b459485f5\" (UID: \"4b57684d-9ed6-45f7-8584-da0b459485f5\") " Jul 20 21:08:07.263152 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:07.263026 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b57684d-9ed6-45f7-8584-da0b459485f5-trusted-ca-bundle\") pod \"4b57684d-9ed6-45f7-8584-da0b459485f5\" (UID: \"4b57684d-9ed6-45f7-8584-da0b459485f5\") " Jul 20 21:08:07.263152 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:07.263090 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4b57684d-9ed6-45f7-8584-da0b459485f5-oauth-serving-cert\") pod \"4b57684d-9ed6-45f7-8584-da0b459485f5\" (UID: \"4b57684d-9ed6-45f7-8584-da0b459485f5\") " Jul 20 21:08:07.263152 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:07.263115 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4l9w\" (UniqueName: \"kubernetes.io/projected/4b57684d-9ed6-45f7-8584-da0b459485f5-kube-api-access-x4l9w\") pod \"4b57684d-9ed6-45f7-8584-da0b459485f5\" (UID: \"4b57684d-9ed6-45f7-8584-da0b459485f5\") " Jul 20 21:08:07.263429 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:07.263208 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4b57684d-9ed6-45f7-8584-da0b459485f5-console-serving-cert\") pod \"4b57684d-9ed6-45f7-8584-da0b459485f5\" (UID: \"4b57684d-9ed6-45f7-8584-da0b459485f5\") " Jul 20 21:08:07.263429 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:07.263240 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4b57684d-9ed6-45f7-8584-da0b459485f5-console-config\") pod \"4b57684d-9ed6-45f7-8584-da0b459485f5\" (UID: \"4b57684d-9ed6-45f7-8584-da0b459485f5\") " Jul 20 21:08:07.263429 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:07.263386 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b57684d-9ed6-45f7-8584-da0b459485f5-service-ca" (OuterVolumeSpecName: "service-ca") pod "4b57684d-9ed6-45f7-8584-da0b459485f5" (UID: "4b57684d-9ed6-45f7-8584-da0b459485f5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 21:08:07.264069 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:07.263606 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b57684d-9ed6-45f7-8584-da0b459485f5-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4b57684d-9ed6-45f7-8584-da0b459485f5" (UID: "4b57684d-9ed6-45f7-8584-da0b459485f5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 21:08:07.264069 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:07.263623 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4b57684d-9ed6-45f7-8584-da0b459485f5-service-ca\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:08:07.264069 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:07.263610 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b57684d-9ed6-45f7-8584-da0b459485f5-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4b57684d-9ed6-45f7-8584-da0b459485f5" (UID: "4b57684d-9ed6-45f7-8584-da0b459485f5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 21:08:07.264069 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:07.263723 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b57684d-9ed6-45f7-8584-da0b459485f5-console-config" (OuterVolumeSpecName: "console-config") pod "4b57684d-9ed6-45f7-8584-da0b459485f5" (UID: "4b57684d-9ed6-45f7-8584-da0b459485f5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 21:08:07.265814 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:07.265760 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b57684d-9ed6-45f7-8584-da0b459485f5-kube-api-access-x4l9w" (OuterVolumeSpecName: "kube-api-access-x4l9w") pod "4b57684d-9ed6-45f7-8584-da0b459485f5" (UID: "4b57684d-9ed6-45f7-8584-da0b459485f5"). InnerVolumeSpecName "kube-api-access-x4l9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 21:08:07.265814 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:07.265791 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b57684d-9ed6-45f7-8584-da0b459485f5-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4b57684d-9ed6-45f7-8584-da0b459485f5" (UID: "4b57684d-9ed6-45f7-8584-da0b459485f5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 21:08:07.275207 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:07.266381 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b57684d-9ed6-45f7-8584-da0b459485f5-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4b57684d-9ed6-45f7-8584-da0b459485f5" (UID: "4b57684d-9ed6-45f7-8584-da0b459485f5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 21:08:07.301265 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:07.301233 2562 generic.go:356] "Generic (PLEG): container finished" podID="4b57684d-9ed6-45f7-8584-da0b459485f5" containerID="89507cc5e5a57c514fa0d983377d074e329d73b78a33a2dbcc5025167d4776ff" exitCode=0 Jul 20 21:08:07.301457 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:07.301295 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bbdc6cd7f-c87gf" Jul 20 21:08:07.301457 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:07.301319 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bbdc6cd7f-c87gf" event={"ID":"4b57684d-9ed6-45f7-8584-da0b459485f5","Type":"ContainerDied","Data":"89507cc5e5a57c514fa0d983377d074e329d73b78a33a2dbcc5025167d4776ff"} Jul 20 21:08:07.301457 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:07.301378 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bbdc6cd7f-c87gf" event={"ID":"4b57684d-9ed6-45f7-8584-da0b459485f5","Type":"ContainerDied","Data":"378ba216620133424fc826fa067ab495b224396d20a3bad74fc28914537971dd"} Jul 20 21:08:07.301457 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:07.301411 2562 scope.go:117] "RemoveContainer" containerID="89507cc5e5a57c514fa0d983377d074e329d73b78a33a2dbcc5025167d4776ff" Jul 20 21:08:07.309721 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:07.309703 2562 scope.go:117] "RemoveContainer" containerID="89507cc5e5a57c514fa0d983377d074e329d73b78a33a2dbcc5025167d4776ff" Jul 20 21:08:07.310022 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:08:07.309994 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89507cc5e5a57c514fa0d983377d074e329d73b78a33a2dbcc5025167d4776ff\": container with ID starting with 89507cc5e5a57c514fa0d983377d074e329d73b78a33a2dbcc5025167d4776ff not found: ID does not exist" containerID="89507cc5e5a57c514fa0d983377d074e329d73b78a33a2dbcc5025167d4776ff" Jul 20 21:08:07.310088 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:07.310031 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89507cc5e5a57c514fa0d983377d074e329d73b78a33a2dbcc5025167d4776ff"} err="failed to get container status \"89507cc5e5a57c514fa0d983377d074e329d73b78a33a2dbcc5025167d4776ff\": rpc error: code = NotFound desc = could not find container \"89507cc5e5a57c514fa0d983377d074e329d73b78a33a2dbcc5025167d4776ff\": container with ID starting with 89507cc5e5a57c514fa0d983377d074e329d73b78a33a2dbcc5025167d4776ff not found: ID does not exist" Jul 20 21:08:07.318735 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:07.318708 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bbdc6cd7f-c87gf"] Jul 20 21:08:07.323531 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:07.323505 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5bbdc6cd7f-c87gf"] Jul 20 21:08:07.364691 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:07.364657 2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4b57684d-9ed6-45f7-8584-da0b459485f5-oauth-serving-cert\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:08:07.364691 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:07.364686 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4l9w\" (UniqueName: \"kubernetes.io/projected/4b57684d-9ed6-45f7-8584-da0b459485f5-kube-api-access-x4l9w\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:08:07.364691 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:07.364697 2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4b57684d-9ed6-45f7-8584-da0b459485f5-console-serving-cert\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:08:07.364913 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:07.364706 2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4b57684d-9ed6-45f7-8584-da0b459485f5-console-config\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:08:07.364913 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:07.364716 2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4b57684d-9ed6-45f7-8584-da0b459485f5-console-oauth-config\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:08:07.364913 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:07.364724 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b57684d-9ed6-45f7-8584-da0b459485f5-trusted-ca-bundle\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:08:09.268997 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:09.268961 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b57684d-9ed6-45f7-8584-da0b459485f5" path="/var/lib/kubelet/pods/4b57684d-9ed6-45f7-8584-da0b459485f5/volumes" Jul 20 21:08:10.756297 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:10.756266 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:08:10.825824 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:10.825787 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:08:14.011959 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:14.011930 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-77c4f499cc-gvnlm" Jul 20 21:08:14.015499 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:14.015481 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-77c4f499cc-gvnlm" Jul 20 21:08:17.897230 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:17.897168 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-75c69c9dc9-vl49s" podUID="280d1cb0-534d-49b9-b267-7882dc079daa" containerName="console" containerID="cri-o://14691d1ad9177fab64de55ce9203ad998575b9f0b222c99d36c982a4f94f7953" gracePeriod=15 Jul 20 21:08:18.139206 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:18.139179 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75c69c9dc9-vl49s" Jul 20 21:08:18.265969 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:18.265933 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/280d1cb0-534d-49b9-b267-7882dc079daa-console-config\") pod \"280d1cb0-534d-49b9-b267-7882dc079daa\" (UID: \"280d1cb0-534d-49b9-b267-7882dc079daa\") " Jul 20 21:08:18.266191 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:18.265983 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/280d1cb0-534d-49b9-b267-7882dc079daa-console-serving-cert\") pod \"280d1cb0-534d-49b9-b267-7882dc079daa\" (UID: \"280d1cb0-534d-49b9-b267-7882dc079daa\") " Jul 20 21:08:18.266191 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:18.266020 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/280d1cb0-534d-49b9-b267-7882dc079daa-service-ca\") pod \"280d1cb0-534d-49b9-b267-7882dc079daa\" (UID: \"280d1cb0-534d-49b9-b267-7882dc079daa\") " Jul 20 21:08:18.266191 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:18.266099 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6g94n\" (UniqueName: \"kubernetes.io/projected/280d1cb0-534d-49b9-b267-7882dc079daa-kube-api-access-6g94n\") pod \"280d1cb0-534d-49b9-b267-7882dc079daa\" (UID: \"280d1cb0-534d-49b9-b267-7882dc079daa\") " Jul 20 21:08:18.266191 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:18.266127 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/280d1cb0-534d-49b9-b267-7882dc079daa-console-oauth-config\") pod \"280d1cb0-534d-49b9-b267-7882dc079daa\" (UID: \"280d1cb0-534d-49b9-b267-7882dc079daa\") " Jul 20 21:08:18.266402 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:18.266211 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/280d1cb0-534d-49b9-b267-7882dc079daa-oauth-serving-cert\") pod \"280d1cb0-534d-49b9-b267-7882dc079daa\" (UID: \"280d1cb0-534d-49b9-b267-7882dc079daa\") " Jul 20 21:08:18.266455 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:18.266390 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/280d1cb0-534d-49b9-b267-7882dc079daa-console-config" (OuterVolumeSpecName: "console-config") pod "280d1cb0-534d-49b9-b267-7882dc079daa" (UID: "280d1cb0-534d-49b9-b267-7882dc079daa"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 21:08:18.266506 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:18.266478 2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/280d1cb0-534d-49b9-b267-7882dc079daa-console-config\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:08:18.266506 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:18.266480 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/280d1cb0-534d-49b9-b267-7882dc079daa-service-ca" (OuterVolumeSpecName: "service-ca") pod "280d1cb0-534d-49b9-b267-7882dc079daa" (UID: "280d1cb0-534d-49b9-b267-7882dc079daa"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 21:08:18.266674 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:18.266646 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/280d1cb0-534d-49b9-b267-7882dc079daa-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "280d1cb0-534d-49b9-b267-7882dc079daa" (UID: "280d1cb0-534d-49b9-b267-7882dc079daa"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 21:08:18.268348 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:18.268322 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/280d1cb0-534d-49b9-b267-7882dc079daa-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "280d1cb0-534d-49b9-b267-7882dc079daa" (UID: "280d1cb0-534d-49b9-b267-7882dc079daa"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 21:08:18.268348 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:18.268330 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/280d1cb0-534d-49b9-b267-7882dc079daa-kube-api-access-6g94n" (OuterVolumeSpecName: "kube-api-access-6g94n") pod "280d1cb0-534d-49b9-b267-7882dc079daa" (UID: "280d1cb0-534d-49b9-b267-7882dc079daa"). InnerVolumeSpecName "kube-api-access-6g94n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 21:08:18.268479 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:18.268346 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/280d1cb0-534d-49b9-b267-7882dc079daa-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "280d1cb0-534d-49b9-b267-7882dc079daa" (UID: "280d1cb0-534d-49b9-b267-7882dc079daa"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 21:08:18.347563 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:18.347531 2562 generic.go:356] "Generic (PLEG): container finished" podID="280d1cb0-534d-49b9-b267-7882dc079daa" containerID="14691d1ad9177fab64de55ce9203ad998575b9f0b222c99d36c982a4f94f7953" exitCode=0 Jul 20 21:08:18.347736 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:18.347576 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75c69c9dc9-vl49s" Jul 20 21:08:18.347736 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:18.347616 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75c69c9dc9-vl49s" event={"ID":"280d1cb0-534d-49b9-b267-7882dc079daa","Type":"ContainerDied","Data":"14691d1ad9177fab64de55ce9203ad998575b9f0b222c99d36c982a4f94f7953"} Jul 20 21:08:18.347736 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:18.347682 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75c69c9dc9-vl49s" event={"ID":"280d1cb0-534d-49b9-b267-7882dc079daa","Type":"ContainerDied","Data":"a65a60a0e5ba5df2ee2580723913c2102a7afe969d25db5b5686673a4ddcb2da"} Jul 20 21:08:18.347736 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:18.347704 2562 scope.go:117] "RemoveContainer" containerID="14691d1ad9177fab64de55ce9203ad998575b9f0b222c99d36c982a4f94f7953" Jul 20 21:08:18.356588 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:18.356570 2562 scope.go:117] "RemoveContainer" containerID="14691d1ad9177fab64de55ce9203ad998575b9f0b222c99d36c982a4f94f7953" Jul 20 21:08:18.356868 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:08:18.356849 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14691d1ad9177fab64de55ce9203ad998575b9f0b222c99d36c982a4f94f7953\": container with ID starting with 14691d1ad9177fab64de55ce9203ad998575b9f0b222c99d36c982a4f94f7953 not found: ID does not exist" containerID="14691d1ad9177fab64de55ce9203ad998575b9f0b222c99d36c982a4f94f7953" Jul 20 21:08:18.356914 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:18.356878 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14691d1ad9177fab64de55ce9203ad998575b9f0b222c99d36c982a4f94f7953"} err="failed to get container status \"14691d1ad9177fab64de55ce9203ad998575b9f0b222c99d36c982a4f94f7953\": rpc error: code = NotFound desc = could not find container \"14691d1ad9177fab64de55ce9203ad998575b9f0b222c99d36c982a4f94f7953\": container with ID starting with 14691d1ad9177fab64de55ce9203ad998575b9f0b222c99d36c982a4f94f7953 not found: ID does not exist" Jul 20 21:08:18.367035 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:18.367002 2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/280d1cb0-534d-49b9-b267-7882dc079daa-console-serving-cert\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:08:18.367035 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:18.367037 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/280d1cb0-534d-49b9-b267-7882dc079daa-service-ca\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:08:18.367239 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:18.367054 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g94n\" (UniqueName: \"kubernetes.io/projected/280d1cb0-534d-49b9-b267-7882dc079daa-kube-api-access-6g94n\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:08:18.367239 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:18.367067 2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/280d1cb0-534d-49b9-b267-7882dc079daa-console-oauth-config\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:08:18.367239 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:18.367079 2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/280d1cb0-534d-49b9-b267-7882dc079daa-oauth-serving-cert\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:08:18.368487 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:18.368464 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75c69c9dc9-vl49s"] Jul 20 21:08:18.374901 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:18.374875 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-75c69c9dc9-vl49s"] Jul 20 21:08:19.269660 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:19.269625 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="280d1cb0-534d-49b9-b267-7882dc079daa" path="/var/lib/kubelet/pods/280d1cb0-534d-49b9-b267-7882dc079daa/volumes" Jul 20 21:08:29.397327 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:29.397289 2562 generic.go:356] "Generic (PLEG): container finished" podID="a002a221-d9f5-42b1-b07b-4b6e946cf5c5" containerID="bbb35fdedc69040c786987b4b4dd9bf29df5ff9ad1693cccdd0f4dfd65302f04" exitCode=0 Jul 20 21:08:29.397709 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:29.397361 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-lv5br" event={"ID":"a002a221-d9f5-42b1-b07b-4b6e946cf5c5","Type":"ContainerDied","Data":"bbb35fdedc69040c786987b4b4dd9bf29df5ff9ad1693cccdd0f4dfd65302f04"} Jul 20 21:08:29.397709 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:29.397698 2562 scope.go:117] "RemoveContainer" containerID="bbb35fdedc69040c786987b4b4dd9bf29df5ff9ad1693cccdd0f4dfd65302f04" Jul 20 21:08:30.402801 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:30.402767 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-lv5br" event={"ID":"a002a221-d9f5-42b1-b07b-4b6e946cf5c5","Type":"ContainerStarted","Data":"0d309bca81f3bf1f31454e9594390911d173c3ecbabf8e5fd5baf48ce5f9a650"} Jul 20 21:08:35.934042 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:35.933955 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:35.953509 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:35.953481 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:36.118412 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:36.118377 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:08:36.219182 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:36.219093 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:08:36.451065 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:36.451038 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:41.823513 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:41.823476 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:08:45.470659 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:45.470623 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:08:45.542111 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:45.542081 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:08:47.175935 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:47.175895 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:08:47.236535 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:47.236506 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:08:53.969896 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:53.969847 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 20 21:08:53.970473 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:53.970153 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b4dd4ce8-4f27-44f8-b790-66afc1d0e273" containerName="prometheus" containerID="cri-o://3ea33f05351732d65f49304f67a74070a95c3e524e3892b1b197f1213fb12a8f" gracePeriod=600 Jul 20 21:08:53.970473 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:53.970177 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b4dd4ce8-4f27-44f8-b790-66afc1d0e273" containerName="kube-rbac-proxy" containerID="cri-o://a55527356515053396018dc3293569fd7f37746763196bfa65f3f5d294686785" gracePeriod=600 Jul 20 21:08:53.970473 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:53.970210 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b4dd4ce8-4f27-44f8-b790-66afc1d0e273" containerName="config-reloader" containerID="cri-o://e6aad72ba73cd8452e8fd0e4182ac4b8a82f41d5367d01d7a1d1a5295237079c" gracePeriod=600 Jul 20 21:08:53.970473 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:53.970235 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b4dd4ce8-4f27-44f8-b790-66afc1d0e273" containerName="kube-rbac-proxy-web" containerID="cri-o://5084e80170cb16afece338654b6fc838047c3e7fc692e4a69bc60fa2b4a9c54d" gracePeriod=600 Jul 20 21:08:53.970473 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:53.970265 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b4dd4ce8-4f27-44f8-b790-66afc1d0e273" containerName="kube-rbac-proxy-thanos" containerID="cri-o://2fb8053d399a787aebf23a8ab2dfc2caaee4c0a60d3439e926da4094ad35e59c" gracePeriod=600 Jul 20 21:08:53.970473 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:53.970205 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b4dd4ce8-4f27-44f8-b790-66afc1d0e273" containerName="thanos-sidecar" containerID="cri-o://3d174c39ac865c56b23aa9724d3d0ab339670a8db7d95a4b90a14eeec2e662e5" gracePeriod=600 Jul 20 21:08:54.508521 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:54.508487 2562 generic.go:356] "Generic (PLEG): container finished" podID="b4dd4ce8-4f27-44f8-b790-66afc1d0e273" containerID="2fb8053d399a787aebf23a8ab2dfc2caaee4c0a60d3439e926da4094ad35e59c" exitCode=0 Jul 20 21:08:54.508521 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:54.508512 2562 generic.go:356] "Generic (PLEG): container finished" podID="b4dd4ce8-4f27-44f8-b790-66afc1d0e273" containerID="a55527356515053396018dc3293569fd7f37746763196bfa65f3f5d294686785" exitCode=0 Jul 20 21:08:54.508521 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:54.508519 2562 generic.go:356] "Generic (PLEG): container finished" podID="b4dd4ce8-4f27-44f8-b790-66afc1d0e273" containerID="3d174c39ac865c56b23aa9724d3d0ab339670a8db7d95a4b90a14eeec2e662e5" exitCode=0 Jul 20 21:08:54.508521 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:54.508524 2562 generic.go:356] "Generic (PLEG): container finished" podID="b4dd4ce8-4f27-44f8-b790-66afc1d0e273" containerID="e6aad72ba73cd8452e8fd0e4182ac4b8a82f41d5367d01d7a1d1a5295237079c" exitCode=0 Jul 20 21:08:54.508521 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:54.508530 2562 generic.go:356] "Generic (PLEG): container finished" podID="b4dd4ce8-4f27-44f8-b790-66afc1d0e273" containerID="3ea33f05351732d65f49304f67a74070a95c3e524e3892b1b197f1213fb12a8f" exitCode=0 Jul 20 21:08:54.508818 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:54.508557 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4dd4ce8-4f27-44f8-b790-66afc1d0e273","Type":"ContainerDied","Data":"2fb8053d399a787aebf23a8ab2dfc2caaee4c0a60d3439e926da4094ad35e59c"} Jul 20 21:08:54.508818 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:54.508619 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4dd4ce8-4f27-44f8-b790-66afc1d0e273","Type":"ContainerDied","Data":"a55527356515053396018dc3293569fd7f37746763196bfa65f3f5d294686785"} Jul 20 21:08:54.508818 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:54.508648 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4dd4ce8-4f27-44f8-b790-66afc1d0e273","Type":"ContainerDied","Data":"3d174c39ac865c56b23aa9724d3d0ab339670a8db7d95a4b90a14eeec2e662e5"} Jul 20 21:08:54.508818 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:54.508672 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4dd4ce8-4f27-44f8-b790-66afc1d0e273","Type":"ContainerDied","Data":"e6aad72ba73cd8452e8fd0e4182ac4b8a82f41d5367d01d7a1d1a5295237079c"} Jul 20 21:08:54.508818 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:54.508694 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4dd4ce8-4f27-44f8-b790-66afc1d0e273","Type":"ContainerDied","Data":"3ea33f05351732d65f49304f67a74070a95c3e524e3892b1b197f1213fb12a8f"} Jul 20 21:08:55.076070 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.076033 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f09f6b9-6ed0-4165-bc40-706eb42f8f0d-metrics-certs\") pod \"network-metrics-daemon-t79t8\" (UID: \"0f09f6b9-6ed0-4165-bc40-706eb42f8f0d\") " pod="openshift-multus/network-metrics-daemon-t79t8" Jul 20 21:08:55.078128 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.078103 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f09f6b9-6ed0-4165-bc40-706eb42f8f0d-metrics-certs\") pod \"network-metrics-daemon-t79t8\" (UID: \"0f09f6b9-6ed0-4165-bc40-706eb42f8f0d\") " pod="openshift-multus/network-metrics-daemon-t79t8" Jul 20 21:08:55.208191 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.208165 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.218500 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.218472 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:08:55.277361 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.277332 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-secret-grpc-tls\") pod \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " Jul 20 21:08:55.277552 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.277373 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-secret-kube-rbac-proxy\") pod \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " Jul 20 21:08:55.277552 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.277421 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-prometheus-k8s-rulefiles-1\") pod \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " Jul 20 21:08:55.277552 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.277447 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-web-config\") pod \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " Jul 20 21:08:55.277552 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.277483 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-prometheus-k8s-rulefiles-0\") pod \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " Jul 20 21:08:55.277552 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.277510 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " Jul 20 21:08:55.277837 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.277799 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "b4dd4ce8-4f27-44f8-b790-66afc1d0e273" (UID: "b4dd4ce8-4f27-44f8-b790-66afc1d0e273"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 21:08:55.277895 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.277872 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " Jul 20 21:08:55.277949 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.277906 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-prometheus-trusted-ca-bundle\") pod \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " Jul 20 21:08:55.277949 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.277930 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sz56g\" (UniqueName: \"kubernetes.io/projected/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-kube-api-access-sz56g\") pod \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " Jul 20 21:08:55.278061 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.277958 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-configmap-serving-certs-ca-bundle\") pod \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " Jul 20 21:08:55.278061 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.278026 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-config-out\") pod \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " Jul 20 21:08:55.278181 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.278063 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-configmap-kubelet-serving-ca-bundle\") pod \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " Jul 20 21:08:55.278181 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.278099 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-prometheus-k8s-db\") pod \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " Jul 20 21:08:55.278181 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.278128 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-secret-metrics-client-certs\") pod \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " Jul 20 21:08:55.278181 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.278168 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-tls-assets\") pod \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " Jul 20 21:08:55.278393 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.278206 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-config\") pod \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " Jul 20 21:08:55.278393 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.278237 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-thanos-prometheus-http-client-file\") pod \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " Jul 20 21:08:55.278393 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.278275 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-configmap-metrics-client-ca\") pod \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " Jul 20 21:08:55.278393 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.278313 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-secret-prometheus-k8s-tls\") pod \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " Jul 20 21:08:55.278393 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.278350 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-prometheus-k8s-rulefiles-2\") pod \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\" (UID: \"b4dd4ce8-4f27-44f8-b790-66afc1d0e273\") " Jul 20 21:08:55.278688 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.278620 2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:08:55.279010 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.278921 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "b4dd4ce8-4f27-44f8-b790-66afc1d0e273" (UID: "b4dd4ce8-4f27-44f8-b790-66afc1d0e273"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 21:08:55.279751 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.279517 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "b4dd4ce8-4f27-44f8-b790-66afc1d0e273" (UID: "b4dd4ce8-4f27-44f8-b790-66afc1d0e273"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 21:08:55.279903 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.279314 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "b4dd4ce8-4f27-44f8-b790-66afc1d0e273" (UID: "b4dd4ce8-4f27-44f8-b790-66afc1d0e273"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 21:08:55.280333 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.280309 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "b4dd4ce8-4f27-44f8-b790-66afc1d0e273" (UID: "b4dd4ce8-4f27-44f8-b790-66afc1d0e273"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 21:08:55.280598 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.280351 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "b4dd4ce8-4f27-44f8-b790-66afc1d0e273" (UID: "b4dd4ce8-4f27-44f8-b790-66afc1d0e273"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 21:08:55.280598 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.280468 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "b4dd4ce8-4f27-44f8-b790-66afc1d0e273" (UID: "b4dd4ce8-4f27-44f8-b790-66afc1d0e273"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 21:08:55.280799 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.280622 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "b4dd4ce8-4f27-44f8-b790-66afc1d0e273" (UID: "b4dd4ce8-4f27-44f8-b790-66afc1d0e273"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 21:08:55.280859 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.280815 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "b4dd4ce8-4f27-44f8-b790-66afc1d0e273" (UID: "b4dd4ce8-4f27-44f8-b790-66afc1d0e273"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 21:08:55.283078 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.282988 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-config" (OuterVolumeSpecName: "config") pod "b4dd4ce8-4f27-44f8-b790-66afc1d0e273" (UID: "b4dd4ce8-4f27-44f8-b790-66afc1d0e273"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 21:08:55.283078 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.282989 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-kube-api-access-sz56g" (OuterVolumeSpecName: "kube-api-access-sz56g") pod "b4dd4ce8-4f27-44f8-b790-66afc1d0e273" (UID: "b4dd4ce8-4f27-44f8-b790-66afc1d0e273"). InnerVolumeSpecName "kube-api-access-sz56g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 21:08:55.283078 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.283050 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-config-out" (OuterVolumeSpecName: "config-out") pod "b4dd4ce8-4f27-44f8-b790-66afc1d0e273" (UID: "b4dd4ce8-4f27-44f8-b790-66afc1d0e273"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 21:08:55.283078 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.283073 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "b4dd4ce8-4f27-44f8-b790-66afc1d0e273" (UID: "b4dd4ce8-4f27-44f8-b790-66afc1d0e273"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 21:08:55.283525 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.283501 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "b4dd4ce8-4f27-44f8-b790-66afc1d0e273" (UID: "b4dd4ce8-4f27-44f8-b790-66afc1d0e273"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 21:08:55.283828 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.283791 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "b4dd4ce8-4f27-44f8-b790-66afc1d0e273" (UID: "b4dd4ce8-4f27-44f8-b790-66afc1d0e273"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 21:08:55.283930 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.283832 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "b4dd4ce8-4f27-44f8-b790-66afc1d0e273" (UID: "b4dd4ce8-4f27-44f8-b790-66afc1d0e273"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 21:08:55.283994 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.283933 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "b4dd4ce8-4f27-44f8-b790-66afc1d0e273" (UID: "b4dd4ce8-4f27-44f8-b790-66afc1d0e273"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 21:08:55.284215 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.284197 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "b4dd4ce8-4f27-44f8-b790-66afc1d0e273" (UID: "b4dd4ce8-4f27-44f8-b790-66afc1d0e273"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 21:08:55.284912 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.284885 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "b4dd4ce8-4f27-44f8-b790-66afc1d0e273" (UID: "b4dd4ce8-4f27-44f8-b790-66afc1d0e273"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 21:08:55.294619 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.294592 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-web-config" (OuterVolumeSpecName: "web-config") pod "b4dd4ce8-4f27-44f8-b790-66afc1d0e273" (UID: "b4dd4ce8-4f27-44f8-b790-66afc1d0e273"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 21:08:55.322479 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.322395 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:08:55.368247 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.368224 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hpjcj\"" Jul 20 21:08:55.376493 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.376466 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t79t8" Jul 20 21:08:55.379060 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.378980 2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:08:55.379060 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.379010 2562 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:08:55.379060 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.379026 2562 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:08:55.379060 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.379041 2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-prometheus-trusted-ca-bundle\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:08:55.379060 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.379054 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz56g\" (UniqueName: \"kubernetes.io/projected/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-kube-api-access-sz56g\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:08:55.379459 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.379069 2562 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:08:55.379459 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.379086 2562 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-config-out\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:08:55.379459 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.379098 2562 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:08:55.379459 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.379110 2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-prometheus-k8s-db\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:08:55.379459 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.379123 2562 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-secret-metrics-client-certs\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:08:55.379459 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.379245 2562 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-tls-assets\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:08:55.379459 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.379255 2562 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-config\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:08:55.379459 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.379265 2562 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-thanos-prometheus-http-client-file\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:08:55.379459 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.379273 2562 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-configmap-metrics-client-ca\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:08:55.379459 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.379286 2562 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-secret-prometheus-k8s-tls\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:08:55.379459 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.379299 2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:08:55.379459 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.379308 2562 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-secret-grpc-tls\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:08:55.379459 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.379316 2562 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-secret-kube-rbac-proxy\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:08:55.379459 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.379325 2562 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b4dd4ce8-4f27-44f8-b790-66afc1d0e273-web-config\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:08:55.501601 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.501548 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-t79t8"] Jul 20 21:08:55.503762 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:08:55.503734 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f09f6b9_6ed0_4165_bc40_706eb42f8f0d.slice/crio-c478e0baf9282290bd5346c7b1f420e35e9458215192fdf90d30d051f3ba6666 WatchSource:0}: Error finding container c478e0baf9282290bd5346c7b1f420e35e9458215192fdf90d30d051f3ba6666: Status 404 returned error can't find the container with id c478e0baf9282290bd5346c7b1f420e35e9458215192fdf90d30d051f3ba6666 Jul 20 21:08:55.505523 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.505505 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 21:08:55.515394 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.515369 2562 generic.go:356] "Generic (PLEG): container finished" podID="b4dd4ce8-4f27-44f8-b790-66afc1d0e273" containerID="5084e80170cb16afece338654b6fc838047c3e7fc692e4a69bc60fa2b4a9c54d" exitCode=0 Jul 20 21:08:55.515492 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.515442 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.515492 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.515449 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4dd4ce8-4f27-44f8-b790-66afc1d0e273","Type":"ContainerDied","Data":"5084e80170cb16afece338654b6fc838047c3e7fc692e4a69bc60fa2b4a9c54d"} Jul 20 21:08:55.515623 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.515522 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4dd4ce8-4f27-44f8-b790-66afc1d0e273","Type":"ContainerDied","Data":"287003b5561d14d1a620354f3f268e6d860a83d1fb34843b56ade69c81988387"} Jul 20 21:08:55.515623 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.515572 2562 scope.go:117] "RemoveContainer" containerID="2fb8053d399a787aebf23a8ab2dfc2caaee4c0a60d3439e926da4094ad35e59c" Jul 20 21:08:55.516793 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.516770 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t79t8" event={"ID":"0f09f6b9-6ed0-4165-bc40-706eb42f8f0d","Type":"ContainerStarted","Data":"c478e0baf9282290bd5346c7b1f420e35e9458215192fdf90d30d051f3ba6666"} Jul 20 21:08:55.523817 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.523747 2562 scope.go:117] "RemoveContainer" containerID="a55527356515053396018dc3293569fd7f37746763196bfa65f3f5d294686785" Jul 20 21:08:55.530877 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.530861 2562 scope.go:117] "RemoveContainer" containerID="5084e80170cb16afece338654b6fc838047c3e7fc692e4a69bc60fa2b4a9c54d" Jul 20 21:08:55.539211 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.539189 2562 scope.go:117] "RemoveContainer" containerID="3d174c39ac865c56b23aa9724d3d0ab339670a8db7d95a4b90a14eeec2e662e5" Jul 20 21:08:55.544402 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.544378 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 20 21:08:55.546869 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.546830 2562 scope.go:117] "RemoveContainer" containerID="e6aad72ba73cd8452e8fd0e4182ac4b8a82f41d5367d01d7a1d1a5295237079c" Jul 20 21:08:55.549179 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.549129 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 20 21:08:55.554000 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.553977 2562 scope.go:117] "RemoveContainer" containerID="3ea33f05351732d65f49304f67a74070a95c3e524e3892b1b197f1213fb12a8f" Jul 20 21:08:55.561252 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.561231 2562 scope.go:117] "RemoveContainer" containerID="144911b5b9578f85d42ea90f280fa17c3af03ea7ca5a916fb0e043e508502988" Jul 20 21:08:55.568218 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.568197 2562 scope.go:117] "RemoveContainer" containerID="2fb8053d399a787aebf23a8ab2dfc2caaee4c0a60d3439e926da4094ad35e59c" Jul 20 21:08:55.568488 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:08:55.568472 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fb8053d399a787aebf23a8ab2dfc2caaee4c0a60d3439e926da4094ad35e59c\": container with ID starting with 2fb8053d399a787aebf23a8ab2dfc2caaee4c0a60d3439e926da4094ad35e59c not found: ID does not exist" containerID="2fb8053d399a787aebf23a8ab2dfc2caaee4c0a60d3439e926da4094ad35e59c" Jul 20 21:08:55.568548 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.568496 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fb8053d399a787aebf23a8ab2dfc2caaee4c0a60d3439e926da4094ad35e59c"} err="failed to get container status \"2fb8053d399a787aebf23a8ab2dfc2caaee4c0a60d3439e926da4094ad35e59c\": rpc error: code = NotFound desc = could not find container \"2fb8053d399a787aebf23a8ab2dfc2caaee4c0a60d3439e926da4094ad35e59c\": container with ID starting with 2fb8053d399a787aebf23a8ab2dfc2caaee4c0a60d3439e926da4094ad35e59c not found: ID does not exist" Jul 20 21:08:55.568548 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.568514 2562 scope.go:117] "RemoveContainer" containerID="a55527356515053396018dc3293569fd7f37746763196bfa65f3f5d294686785" Jul 20 21:08:55.568736 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:08:55.568723 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a55527356515053396018dc3293569fd7f37746763196bfa65f3f5d294686785\": container with ID starting with a55527356515053396018dc3293569fd7f37746763196bfa65f3f5d294686785 not found: ID does not exist" containerID="a55527356515053396018dc3293569fd7f37746763196bfa65f3f5d294686785" Jul 20 21:08:55.568785 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.568739 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a55527356515053396018dc3293569fd7f37746763196bfa65f3f5d294686785"} err="failed to get container status \"a55527356515053396018dc3293569fd7f37746763196bfa65f3f5d294686785\": rpc error: code = NotFound desc = could not find container \"a55527356515053396018dc3293569fd7f37746763196bfa65f3f5d294686785\": container with ID starting with a55527356515053396018dc3293569fd7f37746763196bfa65f3f5d294686785 not found: ID does not exist" Jul 20 21:08:55.568785 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.568750 2562 scope.go:117] "RemoveContainer" containerID="5084e80170cb16afece338654b6fc838047c3e7fc692e4a69bc60fa2b4a9c54d" Jul 20 21:08:55.568942 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:08:55.568928 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5084e80170cb16afece338654b6fc838047c3e7fc692e4a69bc60fa2b4a9c54d\": container with ID starting with 5084e80170cb16afece338654b6fc838047c3e7fc692e4a69bc60fa2b4a9c54d not found: ID does not exist" containerID="5084e80170cb16afece338654b6fc838047c3e7fc692e4a69bc60fa2b4a9c54d" Jul 20 21:08:55.568983 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.568945 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5084e80170cb16afece338654b6fc838047c3e7fc692e4a69bc60fa2b4a9c54d"} err="failed to get container status \"5084e80170cb16afece338654b6fc838047c3e7fc692e4a69bc60fa2b4a9c54d\": rpc error: code = NotFound desc = could not find container \"5084e80170cb16afece338654b6fc838047c3e7fc692e4a69bc60fa2b4a9c54d\": container with ID starting with 5084e80170cb16afece338654b6fc838047c3e7fc692e4a69bc60fa2b4a9c54d not found: ID does not exist" Jul 20 21:08:55.568983 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.568957 2562 scope.go:117] "RemoveContainer" containerID="3d174c39ac865c56b23aa9724d3d0ab339670a8db7d95a4b90a14eeec2e662e5" Jul 20 21:08:55.569181 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:08:55.569162 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d174c39ac865c56b23aa9724d3d0ab339670a8db7d95a4b90a14eeec2e662e5\": container with ID starting with 3d174c39ac865c56b23aa9724d3d0ab339670a8db7d95a4b90a14eeec2e662e5 not found: ID does not exist" containerID="3d174c39ac865c56b23aa9724d3d0ab339670a8db7d95a4b90a14eeec2e662e5" Jul 20 21:08:55.569245 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.569184 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d174c39ac865c56b23aa9724d3d0ab339670a8db7d95a4b90a14eeec2e662e5"} err="failed to get container status \"3d174c39ac865c56b23aa9724d3d0ab339670a8db7d95a4b90a14eeec2e662e5\": rpc error: code = NotFound desc = could not find container \"3d174c39ac865c56b23aa9724d3d0ab339670a8db7d95a4b90a14eeec2e662e5\": container with ID starting with 3d174c39ac865c56b23aa9724d3d0ab339670a8db7d95a4b90a14eeec2e662e5 not found: ID does not exist" Jul 20 21:08:55.569245 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.569199 2562 scope.go:117] "RemoveContainer" containerID="e6aad72ba73cd8452e8fd0e4182ac4b8a82f41d5367d01d7a1d1a5295237079c" Jul 20 21:08:55.569401 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:08:55.569381 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6aad72ba73cd8452e8fd0e4182ac4b8a82f41d5367d01d7a1d1a5295237079c\": container with ID starting with e6aad72ba73cd8452e8fd0e4182ac4b8a82f41d5367d01d7a1d1a5295237079c not found: ID does not exist" containerID="e6aad72ba73cd8452e8fd0e4182ac4b8a82f41d5367d01d7a1d1a5295237079c" Jul 20 21:08:55.569454 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.569407 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6aad72ba73cd8452e8fd0e4182ac4b8a82f41d5367d01d7a1d1a5295237079c"} err="failed to get container status \"e6aad72ba73cd8452e8fd0e4182ac4b8a82f41d5367d01d7a1d1a5295237079c\": rpc error: code = NotFound desc = could not find container \"e6aad72ba73cd8452e8fd0e4182ac4b8a82f41d5367d01d7a1d1a5295237079c\": container with ID starting with e6aad72ba73cd8452e8fd0e4182ac4b8a82f41d5367d01d7a1d1a5295237079c not found: ID does not exist" Jul 20 21:08:55.569454 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.569423 2562 scope.go:117] "RemoveContainer" containerID="3ea33f05351732d65f49304f67a74070a95c3e524e3892b1b197f1213fb12a8f" Jul 20 21:08:55.569623 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:08:55.569608 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ea33f05351732d65f49304f67a74070a95c3e524e3892b1b197f1213fb12a8f\": container with ID starting with 3ea33f05351732d65f49304f67a74070a95c3e524e3892b1b197f1213fb12a8f not found: ID does not exist" containerID="3ea33f05351732d65f49304f67a74070a95c3e524e3892b1b197f1213fb12a8f" Jul 20 21:08:55.569662 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.569626 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ea33f05351732d65f49304f67a74070a95c3e524e3892b1b197f1213fb12a8f"} err="failed to get container status \"3ea33f05351732d65f49304f67a74070a95c3e524e3892b1b197f1213fb12a8f\": rpc error: code = NotFound desc = could not find container \"3ea33f05351732d65f49304f67a74070a95c3e524e3892b1b197f1213fb12a8f\": container with ID starting with 3ea33f05351732d65f49304f67a74070a95c3e524e3892b1b197f1213fb12a8f not found: ID does not exist" Jul 20 21:08:55.569662 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.569638 2562 scope.go:117] "RemoveContainer" containerID="144911b5b9578f85d42ea90f280fa17c3af03ea7ca5a916fb0e043e508502988" Jul 20 21:08:55.569877 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:08:55.569858 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"144911b5b9578f85d42ea90f280fa17c3af03ea7ca5a916fb0e043e508502988\": container with ID starting with 144911b5b9578f85d42ea90f280fa17c3af03ea7ca5a916fb0e043e508502988 not found: ID does not exist" containerID="144911b5b9578f85d42ea90f280fa17c3af03ea7ca5a916fb0e043e508502988" Jul 20 21:08:55.569917 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.569880 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"144911b5b9578f85d42ea90f280fa17c3af03ea7ca5a916fb0e043e508502988"} err="failed to get container status \"144911b5b9578f85d42ea90f280fa17c3af03ea7ca5a916fb0e043e508502988\": rpc error: code = NotFound desc = could not find container \"144911b5b9578f85d42ea90f280fa17c3af03ea7ca5a916fb0e043e508502988\": container with ID starting with 144911b5b9578f85d42ea90f280fa17c3af03ea7ca5a916fb0e043e508502988 not found: ID does not exist" Jul 20 21:08:55.575757 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.575694 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 20 21:08:55.576089 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.576073 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="280d1cb0-534d-49b9-b267-7882dc079daa" containerName="console" Jul 20 21:08:55.576182 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.576090 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="280d1cb0-534d-49b9-b267-7882dc079daa" containerName="console" Jul 20 21:08:55.576182 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.576103 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4dd4ce8-4f27-44f8-b790-66afc1d0e273" containerName="kube-rbac-proxy-thanos" Jul 20 21:08:55.576182 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.576113 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4dd4ce8-4f27-44f8-b790-66afc1d0e273" containerName="kube-rbac-proxy-thanos" Jul 20 21:08:55.576182 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.576126 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4dd4ce8-4f27-44f8-b790-66afc1d0e273" containerName="kube-rbac-proxy-web" Jul 20 21:08:55.576182 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.576150 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4dd4ce8-4f27-44f8-b790-66afc1d0e273" containerName="kube-rbac-proxy-web" Jul 20 21:08:55.576182 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.576164 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4dd4ce8-4f27-44f8-b790-66afc1d0e273" containerName="init-config-reloader" Jul 20 21:08:55.576182 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.576172 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4dd4ce8-4f27-44f8-b790-66afc1d0e273" containerName="init-config-reloader" Jul 20 21:08:55.576519 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.576192 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4dd4ce8-4f27-44f8-b790-66afc1d0e273" containerName="thanos-sidecar" Jul 20 21:08:55.576519 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.576201 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4dd4ce8-4f27-44f8-b790-66afc1d0e273" containerName="thanos-sidecar" Jul 20 21:08:55.576519 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.576213 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e63b84e-6cf9-467e-b114-9682be23c2b9" containerName="registry" Jul 20 21:08:55.576519 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.576221 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e63b84e-6cf9-467e-b114-9682be23c2b9" containerName="registry" Jul 20 21:08:55.576519 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.576231 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b57684d-9ed6-45f7-8584-da0b459485f5" containerName="console" Jul 20 21:08:55.576519 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.576239 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b57684d-9ed6-45f7-8584-da0b459485f5" containerName="console" Jul 20 21:08:55.576519 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.576251 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4dd4ce8-4f27-44f8-b790-66afc1d0e273" containerName="prometheus" Jul 20 21:08:55.576519 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.576258 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4dd4ce8-4f27-44f8-b790-66afc1d0e273" containerName="prometheus" Jul 20 21:08:55.576519 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.576271 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4dd4ce8-4f27-44f8-b790-66afc1d0e273" containerName="config-reloader" Jul 20 21:08:55.576519 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.576279 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4dd4ce8-4f27-44f8-b790-66afc1d0e273" containerName="config-reloader" Jul 20 21:08:55.576519 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.576292 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4dd4ce8-4f27-44f8-b790-66afc1d0e273" containerName="kube-rbac-proxy" Jul 20 21:08:55.576519 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.576301 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4dd4ce8-4f27-44f8-b790-66afc1d0e273" containerName="kube-rbac-proxy" Jul 20 21:08:55.576519 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.576378 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4dd4ce8-4f27-44f8-b790-66afc1d0e273" containerName="thanos-sidecar" Jul 20 21:08:55.576519 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.576390 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4dd4ce8-4f27-44f8-b790-66afc1d0e273" containerName="prometheus" Jul 20 21:08:55.576519 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.576399 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e63b84e-6cf9-467e-b114-9682be23c2b9" containerName="registry" Jul 20 21:08:55.576519 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.576409 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4dd4ce8-4f27-44f8-b790-66afc1d0e273" containerName="config-reloader" Jul 20 21:08:55.576519 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.576416 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4dd4ce8-4f27-44f8-b790-66afc1d0e273" containerName="kube-rbac-proxy-web" Jul 20 21:08:55.576519 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.576427 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="280d1cb0-534d-49b9-b267-7882dc079daa" containerName="console" Jul 20 21:08:55.576519 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.576436 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4dd4ce8-4f27-44f8-b790-66afc1d0e273" containerName="kube-rbac-proxy-thanos" Jul 20 21:08:55.576519 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.576446 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4dd4ce8-4f27-44f8-b790-66afc1d0e273" containerName="kube-rbac-proxy" Jul 20 21:08:55.576519 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.576454 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b57684d-9ed6-45f7-8584-da0b459485f5" containerName="console" Jul 20 21:08:55.579797 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.579778 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.582582 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.582552 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 20 21:08:55.582688 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.582589 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 20 21:08:55.582688 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.582555 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9u1e07mk6t0nk\"" Jul 20 21:08:55.582799 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.582555 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 20 21:08:55.582909 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.582884 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 20 21:08:55.582909 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.582887 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 20 21:08:55.583036 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.582889 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 20 21:08:55.583036 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.582926 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 20 21:08:55.583036 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.582939 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 20 21:08:55.583036 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.582962 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-65242\"" Jul 20 21:08:55.583224 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.583179 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 20 21:08:55.583279 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.583223 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 20 21:08:55.583578 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.583564 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 20 21:08:55.583638 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.583621 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 20 21:08:55.586583 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.586565 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 20 21:08:55.593300 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.591155 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 20 21:08:55.596720 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.596696 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 20 21:08:55.681783 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.681745 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.681783 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.681790 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.681995 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.681809 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-config\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.681995 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.681831 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.681995 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.681866 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.681995 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.681885 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-config-out\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.681995 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.681904 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzjql\" (UniqueName: \"kubernetes.io/projected/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-kube-api-access-jzjql\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.681995 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.681943 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.681995 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.681958 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.681995 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.681982 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.682275 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.682003 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.682275 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.682022 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.682275 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.682038 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-web-config\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.682275 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.682065 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.682275 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.682092 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.682275 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.682107 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.682275 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.682156 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.682275 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.682179 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.682275 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.682197 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.682275 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.682222 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.782798 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.782765 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-config\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.782916 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.782801 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.782916 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.782829 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.782916 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.782849 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-config-out\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.783024 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.782920 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzjql\" (UniqueName: \"kubernetes.io/projected/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-kube-api-access-jzjql\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.783024 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.782951 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.783024 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.782975 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.783194 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.783101 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.783194 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.783127 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.783194 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.783177 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.783354 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.783200 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-web-config\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.783354 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.783220 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.783354 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.783252 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.783354 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.783291 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.783354 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.783336 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.783607 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.783368 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.783607 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.783398 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.783607 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.783440 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.783607 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.783467 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.783607 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.783495 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.783863 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.783645 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.784003 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.783980 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.785752 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.785049 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.785752 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.785371 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.785752 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.785438 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.786593 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.786036 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.786593 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.786484 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.786593 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.786553 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.786593 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.786577 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.787300 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.786956 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-config-out\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.787300 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.787260 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-config\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.787459 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.787389 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.788008 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.787984 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.788123 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.788004 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.788123 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.788030 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.788123 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.788075 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.788325 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.788285 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-web-config\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.788548 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.788528 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.789674 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.789656 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.791663 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.791643 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzjql\" (UniqueName: \"kubernetes.io/projected/319342c3-f33c-4b9b-ac0b-c8ffe0bc312a-kube-api-access-jzjql\") pod \"prometheus-k8s-0\" (UID: \"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:55.895995 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:55.895907 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:08:56.059767 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:56.059691 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 20 21:08:56.062946 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:08:56.062914 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod319342c3_f33c_4b9b_ac0b_c8ffe0bc312a.slice/crio-0108eaeed95d37b67d0544391c8ea937c6db2247e8a57e72ab41e1b1256dc278 WatchSource:0}: Error finding container 0108eaeed95d37b67d0544391c8ea937c6db2247e8a57e72ab41e1b1256dc278: Status 404 returned error can't find the container with id 0108eaeed95d37b67d0544391c8ea937c6db2247e8a57e72ab41e1b1256dc278 Jul 20 21:08:56.523231 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:56.523194 2562 generic.go:356] "Generic (PLEG): container finished" podID="319342c3-f33c-4b9b-ac0b-c8ffe0bc312a" containerID="023cbf52e56bb716d652e4c5a8799af334d155e993f5aba704e635eff0f41b23" exitCode=0 Jul 20 21:08:56.523660 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:56.523356 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a","Type":"ContainerDied","Data":"023cbf52e56bb716d652e4c5a8799af334d155e993f5aba704e635eff0f41b23"} Jul 20 21:08:56.523660 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:56.523429 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a","Type":"ContainerStarted","Data":"0108eaeed95d37b67d0544391c8ea937c6db2247e8a57e72ab41e1b1256dc278"} Jul 20 21:08:56.529210 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:56.529178 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t79t8" event={"ID":"0f09f6b9-6ed0-4165-bc40-706eb42f8f0d","Type":"ContainerStarted","Data":"d9ed529c02ec624f1ae2b86da24592aa40c3870315206902392aff9f66035759"} Jul 20 21:08:57.271999 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:57.271965 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4dd4ce8-4f27-44f8-b790-66afc1d0e273" path="/var/lib/kubelet/pods/b4dd4ce8-4f27-44f8-b790-66afc1d0e273/volumes" Jul 20 21:08:57.411697 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:57.411655 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:08:57.466998 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:57.466948 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:08:57.537462 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:57.537361 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a","Type":"ContainerStarted","Data":"c30222d5c999cd0c627b330896d161639689fd39f50e19cdc04c9f72d26c0d24"} Jul 20 21:08:57.537462 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:57.537452 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a","Type":"ContainerStarted","Data":"0e349e4307e74966ba9e66eda305b84262ca1dab07e32e1910be85821da2135e"} Jul 20 21:08:57.537913 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:57.537491 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a","Type":"ContainerStarted","Data":"daceb5871004a956e35530b017130155150fbb32296349aed2c6d9d2157e5f24"} Jul 20 21:08:57.537913 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:57.537527 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a","Type":"ContainerStarted","Data":"79de5afc323dc43ce462be1990b2df8a8ad7d56c6b3bf46011d94b574b1ac206"} Jul 20 21:08:57.537913 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:57.537571 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a","Type":"ContainerStarted","Data":"13d94a440ab958403344a2e12cd041e01c8d85b5507e61bff01fbca5b5f86ee9"} Jul 20 21:08:57.537913 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:57.537601 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"319342c3-f33c-4b9b-ac0b-c8ffe0bc312a","Type":"ContainerStarted","Data":"28fd576c481237bc87719a5471c1f9fecb6722ab58da4d1923bfb8ccf1de1114"} Jul 20 21:08:57.538939 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:57.538919 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t79t8" event={"ID":"0f09f6b9-6ed0-4165-bc40-706eb42f8f0d","Type":"ContainerStarted","Data":"e04dd9b12a1abd2af4be71e8815791510460b611f07bfa6a60569d89c50aff14"} Jul 20 21:08:57.569454 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:57.569405 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.546956546 podStartE2EDuration="2.569390938s" totalImagesPullingTime="0s" totalInitContainerRuntime="22.434392ms" isStatefulPod=false podCreationTimestamp="2026-07-20 21:08:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:08:57.567016593 +0000 UTC m=+376.892719535" watchObservedRunningTime="2026-07-20 21:08:57.569390938 +0000 UTC m=+376.895093880" Jul 20 21:08:57.682005 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:57.681969 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:08:57.745060 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:57.745030 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:08:58.299959 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:58.299927 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:08:58.357819 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:58.357792 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:08:58.419056 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:08:58.419026 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:09:00.896367 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:09:00.896324 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:09:10.264156 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:09:10.264111 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:09:10.329048 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:09:10.329018 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:09:10.731837 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:09:10.731759 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:09:10.832361 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:09:10.832331 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:09:19.796012 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:09:19.795976 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:09:44.930004 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:09:44.929967 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:09:45.028387 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:09:45.028341 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:09:51.745469 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:09:51.745434 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:09:51.846038 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:09:51.845999 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:09:55.896293 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:09:55.896261 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:09:55.911775 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:09:55.911746 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:09:55.944492 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:09:55.944425 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-t79t8" podStartSLOduration=434.02966483 podStartE2EDuration="7m14.94440401s" totalImagesPullingTime="914.73918ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 21:02:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:08:57.583328041 +0000 UTC m=+376.909030979" watchObservedRunningTime="2026-07-20 21:09:55.94440401 +0000 UTC m=+435.270106955" Jul 20 21:09:56.801304 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:09:56.801275 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 20 21:10:11.941894 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:10:11.941854 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:10:12.874320 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:10:12.874282 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:10:15.575920 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:10:15.575887 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:10:15.676793 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:10:15.676759 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:10:17.448197 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:10:17.448159 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:10:17.549890 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:10:17.549858 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:10:17.823618 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:10:17.823582 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:10:17.933854 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:10:17.933820 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:10:19.449382 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:10:19.449352 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:10:19.832472 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:10:19.832443 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:10:19.945210 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:10:19.945177 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-48gbb"] Jul 20 21:10:19.948552 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:10:19.948533 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-48gbb" Jul 20 21:10:19.951809 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:10:19.951789 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 20 21:10:19.952710 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:10:19.952684 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 20 21:10:19.952819 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:10:19.952742 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-tdhld\"" Jul 20 21:10:19.964549 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:10:19.964522 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-48gbb"] Jul 20 21:10:19.998425 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:10:19.998383 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b847n\" (UniqueName: \"kubernetes.io/projected/18bcabba-756d-4e21-b17d-61ba0ca8a4fe-kube-api-access-b847n\") pod \"cert-manager-cainjector-559cdd8c87-48gbb\" (UID: \"18bcabba-756d-4e21-b17d-61ba0ca8a4fe\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-48gbb" Jul 20 21:10:19.998661 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:10:19.998456 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/18bcabba-756d-4e21-b17d-61ba0ca8a4fe-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-48gbb\" (UID: \"18bcabba-756d-4e21-b17d-61ba0ca8a4fe\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-48gbb" Jul 20 21:10:20.099284 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:10:20.099192 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b847n\" (UniqueName: \"kubernetes.io/projected/18bcabba-756d-4e21-b17d-61ba0ca8a4fe-kube-api-access-b847n\") pod \"cert-manager-cainjector-559cdd8c87-48gbb\" (UID: \"18bcabba-756d-4e21-b17d-61ba0ca8a4fe\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-48gbb" Jul 20 21:10:20.099284 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:10:20.099253 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/18bcabba-756d-4e21-b17d-61ba0ca8a4fe-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-48gbb\" (UID: \"18bcabba-756d-4e21-b17d-61ba0ca8a4fe\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-48gbb" Jul 20 21:10:20.110492 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:10:20.110462 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/18bcabba-756d-4e21-b17d-61ba0ca8a4fe-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-48gbb\" (UID: \"18bcabba-756d-4e21-b17d-61ba0ca8a4fe\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-48gbb" Jul 20 21:10:20.110695 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:10:20.110669 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b847n\" (UniqueName: \"kubernetes.io/projected/18bcabba-756d-4e21-b17d-61ba0ca8a4fe-kube-api-access-b847n\") pod \"cert-manager-cainjector-559cdd8c87-48gbb\" (UID: \"18bcabba-756d-4e21-b17d-61ba0ca8a4fe\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-48gbb" Jul 20 21:10:20.269907 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:10:20.269871 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-48gbb" Jul 20 21:10:20.399185 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:10:20.399145 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-48gbb"] Jul 20 21:10:20.402031 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:10:20.402003 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18bcabba_756d_4e21_b17d_61ba0ca8a4fe.slice/crio-b80fbe6d8d5ca11aad1513fd98b9fc007bf9001834175fadcde1aa2b178a2d47 WatchSource:0}: Error finding container b80fbe6d8d5ca11aad1513fd98b9fc007bf9001834175fadcde1aa2b178a2d47: Status 404 returned error can't find the container with id b80fbe6d8d5ca11aad1513fd98b9fc007bf9001834175fadcde1aa2b178a2d47 Jul 20 21:10:20.889117 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:10:20.889085 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-48gbb" event={"ID":"18bcabba-756d-4e21-b17d-61ba0ca8a4fe","Type":"ContainerStarted","Data":"b80fbe6d8d5ca11aad1513fd98b9fc007bf9001834175fadcde1aa2b178a2d47"} Jul 20 21:10:22.562787 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:10:22.562751 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:10:22.664330 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:10:22.664269 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:10:23.904019 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:10:23.903977 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-48gbb" event={"ID":"18bcabba-756d-4e21-b17d-61ba0ca8a4fe","Type":"ContainerStarted","Data":"5d77ee7cd6552b166865a0cf1dd4b154d54be71d40eefcce55781fbaa04c49a5"} Jul 20 21:10:23.921013 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:10:23.920958 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-48gbb" podStartSLOduration=1.8636576630000001 podStartE2EDuration="4.920942138s" totalImagesPullingTime="3.057284475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 21:10:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:10:23.919861533 +0000 UTC m=+463.245564477" watchObservedRunningTime="2026-07-20 21:10:23.920942138 +0000 UTC m=+463.246645081" Jul 20 21:10:27.193365 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:10:27.193330 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:10:27.302576 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:10:27.302541 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:10:28.246094 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:10:28.246061 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:10:38.200783 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:10:38.200743 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-g65qs"] Jul 20 21:10:38.203956 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:10:38.203934 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-g65qs" Jul 20 21:10:38.206790 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:10:38.206764 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-t7mz5\"" Jul 20 21:10:38.214332 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:10:38.214306 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-g65qs"] Jul 20 21:10:38.250519 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:10:38.250474 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvrmf\" (UniqueName: \"kubernetes.io/projected/350780d1-112a-4df2-ad03-89bc4575bdfc-kube-api-access-rvrmf\") pod \"cert-manager-55b4d954d5-g65qs\" (UID: \"350780d1-112a-4df2-ad03-89bc4575bdfc\") " pod="cert-manager/cert-manager-55b4d954d5-g65qs" Jul 20 21:10:38.250714 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:10:38.250533 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/350780d1-112a-4df2-ad03-89bc4575bdfc-bound-sa-token\") pod \"cert-manager-55b4d954d5-g65qs\" (UID: \"350780d1-112a-4df2-ad03-89bc4575bdfc\") " pod="cert-manager/cert-manager-55b4d954d5-g65qs" Jul 20 21:10:38.351148 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:10:38.351104 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvrmf\" (UniqueName: \"kubernetes.io/projected/350780d1-112a-4df2-ad03-89bc4575bdfc-kube-api-access-rvrmf\") pod \"cert-manager-55b4d954d5-g65qs\" (UID: \"350780d1-112a-4df2-ad03-89bc4575bdfc\") " pod="cert-manager/cert-manager-55b4d954d5-g65qs" Jul 20 21:10:38.351327 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:10:38.351177 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/350780d1-112a-4df2-ad03-89bc4575bdfc-bound-sa-token\") pod \"cert-manager-55b4d954d5-g65qs\" (UID: \"350780d1-112a-4df2-ad03-89bc4575bdfc\") " pod="cert-manager/cert-manager-55b4d954d5-g65qs" Jul 20 21:10:38.359351 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:10:38.359323 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/350780d1-112a-4df2-ad03-89bc4575bdfc-bound-sa-token\") pod \"cert-manager-55b4d954d5-g65qs\" (UID: \"350780d1-112a-4df2-ad03-89bc4575bdfc\") " pod="cert-manager/cert-manager-55b4d954d5-g65qs" Jul 20 21:10:38.359556 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:10:38.359534 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvrmf\" (UniqueName: \"kubernetes.io/projected/350780d1-112a-4df2-ad03-89bc4575bdfc-kube-api-access-rvrmf\") pod \"cert-manager-55b4d954d5-g65qs\" (UID: \"350780d1-112a-4df2-ad03-89bc4575bdfc\") " pod="cert-manager/cert-manager-55b4d954d5-g65qs" Jul 20 21:10:38.513716 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:10:38.513692 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-g65qs" Jul 20 21:10:38.645392 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:10:38.645355 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-g65qs"] Jul 20 21:10:38.648675 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:10:38.648647 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod350780d1_112a_4df2_ad03_89bc4575bdfc.slice/crio-fc0f174da9df09a25ec7c44b54ebb9594f3a10204bce0f7d3329bc77ca02243a WatchSource:0}: Error finding container fc0f174da9df09a25ec7c44b54ebb9594f3a10204bce0f7d3329bc77ca02243a: Status 404 returned error can't find the container with id fc0f174da9df09a25ec7c44b54ebb9594f3a10204bce0f7d3329bc77ca02243a Jul 20 21:10:38.965524 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:10:38.965433 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-g65qs" event={"ID":"350780d1-112a-4df2-ad03-89bc4575bdfc","Type":"ContainerStarted","Data":"b76482c726151482131ccd507b9cbcc705c0f5471849d2e546f0baae00169ed0"} Jul 20 21:10:38.965524 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:10:38.965477 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-g65qs" event={"ID":"350780d1-112a-4df2-ad03-89bc4575bdfc","Type":"ContainerStarted","Data":"fc0f174da9df09a25ec7c44b54ebb9594f3a10204bce0f7d3329bc77ca02243a"} Jul 20 21:10:38.983290 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:10:38.983235 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-g65qs" podStartSLOduration=0.98321929 podStartE2EDuration="983.21929ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 21:10:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:10:38.981462101 +0000 UTC m=+478.307165046" watchObservedRunningTime="2026-07-20 21:10:38.98321929 +0000 UTC m=+478.308922233" Jul 20 21:10:40.290751 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:10:40.290719 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:10:40.412512 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:10:40.412475 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:10:48.010009 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:10:48.009972 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:10:48.104719 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:10:48.104680 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:10:49.860534 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:10:49.860499 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:11:06.052175 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:06.052119 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:11:06.155613 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:06.155575 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:11:06.251252 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:06.251212 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:11:06.251534 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:06.251514 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7475cd49b4-ns279"] Jul 20 21:11:06.253781 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:06.253765 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7475cd49b4-ns279" Jul 20 21:11:06.258019 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:06.257995 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 20 21:11:06.258185 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:06.258027 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 20 21:11:06.258185 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:06.258027 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 20 21:11:06.258185 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:06.258177 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 20 21:11:06.258364 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:06.258034 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-vkcvw\"" Jul 20 21:11:06.258364 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:06.258080 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 20 21:11:06.273630 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:06.273600 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7475cd49b4-ns279"] Jul 20 21:11:06.398596 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:06.398505 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbprr\" (UniqueName: \"kubernetes.io/projected/b047db61-e3ec-4c54-83f9-d14bc3dd38f5-kube-api-access-lbprr\") pod \"lws-controller-manager-7475cd49b4-ns279\" (UID: \"b047db61-e3ec-4c54-83f9-d14bc3dd38f5\") " pod="openshift-lws-operator/lws-controller-manager-7475cd49b4-ns279" Jul 20 21:11:06.398596 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:06.398559 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b047db61-e3ec-4c54-83f9-d14bc3dd38f5-cert\") pod \"lws-controller-manager-7475cd49b4-ns279\" (UID: \"b047db61-e3ec-4c54-83f9-d14bc3dd38f5\") " pod="openshift-lws-operator/lws-controller-manager-7475cd49b4-ns279" Jul 20 21:11:06.398787 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:06.398647 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b047db61-e3ec-4c54-83f9-d14bc3dd38f5-manager-config\") pod \"lws-controller-manager-7475cd49b4-ns279\" (UID: \"b047db61-e3ec-4c54-83f9-d14bc3dd38f5\") " pod="openshift-lws-operator/lws-controller-manager-7475cd49b4-ns279" Jul 20 21:11:06.398787 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:06.398703 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b047db61-e3ec-4c54-83f9-d14bc3dd38f5-metrics-cert\") pod \"lws-controller-manager-7475cd49b4-ns279\" (UID: \"b047db61-e3ec-4c54-83f9-d14bc3dd38f5\") " pod="openshift-lws-operator/lws-controller-manager-7475cd49b4-ns279" Jul 20 21:11:06.500036 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:06.499986 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b047db61-e3ec-4c54-83f9-d14bc3dd38f5-manager-config\") pod \"lws-controller-manager-7475cd49b4-ns279\" (UID: \"b047db61-e3ec-4c54-83f9-d14bc3dd38f5\") " pod="openshift-lws-operator/lws-controller-manager-7475cd49b4-ns279" Jul 20 21:11:06.500269 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:06.500048 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b047db61-e3ec-4c54-83f9-d14bc3dd38f5-metrics-cert\") pod \"lws-controller-manager-7475cd49b4-ns279\" (UID: \"b047db61-e3ec-4c54-83f9-d14bc3dd38f5\") " pod="openshift-lws-operator/lws-controller-manager-7475cd49b4-ns279" Jul 20 21:11:06.500269 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:06.500074 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbprr\" (UniqueName: \"kubernetes.io/projected/b047db61-e3ec-4c54-83f9-d14bc3dd38f5-kube-api-access-lbprr\") pod \"lws-controller-manager-7475cd49b4-ns279\" (UID: \"b047db61-e3ec-4c54-83f9-d14bc3dd38f5\") " pod="openshift-lws-operator/lws-controller-manager-7475cd49b4-ns279" Jul 20 21:11:06.500269 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:06.500105 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b047db61-e3ec-4c54-83f9-d14bc3dd38f5-cert\") pod \"lws-controller-manager-7475cd49b4-ns279\" (UID: \"b047db61-e3ec-4c54-83f9-d14bc3dd38f5\") " pod="openshift-lws-operator/lws-controller-manager-7475cd49b4-ns279" Jul 20 21:11:06.500702 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:06.500675 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b047db61-e3ec-4c54-83f9-d14bc3dd38f5-manager-config\") pod \"lws-controller-manager-7475cd49b4-ns279\" (UID: \"b047db61-e3ec-4c54-83f9-d14bc3dd38f5\") " pod="openshift-lws-operator/lws-controller-manager-7475cd49b4-ns279" Jul 20 21:11:06.502647 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:06.502623 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b047db61-e3ec-4c54-83f9-d14bc3dd38f5-metrics-cert\") pod \"lws-controller-manager-7475cd49b4-ns279\" (UID: \"b047db61-e3ec-4c54-83f9-d14bc3dd38f5\") " pod="openshift-lws-operator/lws-controller-manager-7475cd49b4-ns279" Jul 20 21:11:06.502736 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:06.502696 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b047db61-e3ec-4c54-83f9-d14bc3dd38f5-cert\") pod \"lws-controller-manager-7475cd49b4-ns279\" (UID: \"b047db61-e3ec-4c54-83f9-d14bc3dd38f5\") " pod="openshift-lws-operator/lws-controller-manager-7475cd49b4-ns279" Jul 20 21:11:06.510525 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:06.510497 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbprr\" (UniqueName: \"kubernetes.io/projected/b047db61-e3ec-4c54-83f9-d14bc3dd38f5-kube-api-access-lbprr\") pod \"lws-controller-manager-7475cd49b4-ns279\" (UID: \"b047db61-e3ec-4c54-83f9-d14bc3dd38f5\") " pod="openshift-lws-operator/lws-controller-manager-7475cd49b4-ns279" Jul 20 21:11:06.563567 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:06.563518 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7475cd49b4-ns279" Jul 20 21:11:06.705363 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:06.705337 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7475cd49b4-ns279"] Jul 20 21:11:06.707588 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:11:06.707559 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb047db61_e3ec_4c54_83f9_d14bc3dd38f5.slice/crio-0e83bcffae7fa7c1946bddaa2799083a55639c713caabf67968b7a2976008d46 WatchSource:0}: Error finding container 0e83bcffae7fa7c1946bddaa2799083a55639c713caabf67968b7a2976008d46: Status 404 returned error can't find the container with id 0e83bcffae7fa7c1946bddaa2799083a55639c713caabf67968b7a2976008d46 Jul 20 21:11:06.867393 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:06.867350 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:11:07.080076 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:07.080039 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7475cd49b4-ns279" event={"ID":"b047db61-e3ec-4c54-83f9-d14bc3dd38f5","Type":"ContainerStarted","Data":"0e83bcffae7fa7c1946bddaa2799083a55639c713caabf67968b7a2976008d46"} Jul 20 21:11:07.184727 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:07.184693 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:11:07.289577 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:07.289531 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:11:07.349381 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:07.349294 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-66b5c4b596-9klsp"] Jul 20 21:11:07.353446 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:07.353412 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-66b5c4b596-9klsp" Jul 20 21:11:07.357205 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:07.356941 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 20 21:11:07.357353 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:07.357336 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 20 21:11:07.357425 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:07.357359 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-tc9ds\"" Jul 20 21:11:07.357908 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:07.357795 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 20 21:11:07.358333 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:07.358167 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 20 21:11:07.375014 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:07.374985 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-66b5c4b596-9klsp"] Jul 20 21:11:07.508381 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:07.508308 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9kct\" (UniqueName: \"kubernetes.io/projected/1a9af75b-a8d9-47b3-976a-be47fbe35b4e-kube-api-access-b9kct\") pod \"opendatahub-operator-controller-manager-66b5c4b596-9klsp\" (UID: \"1a9af75b-a8d9-47b3-976a-be47fbe35b4e\") " pod="opendatahub/opendatahub-operator-controller-manager-66b5c4b596-9klsp" Jul 20 21:11:07.508612 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:07.508455 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1a9af75b-a8d9-47b3-976a-be47fbe35b4e-tmp\") pod \"opendatahub-operator-controller-manager-66b5c4b596-9klsp\" (UID: \"1a9af75b-a8d9-47b3-976a-be47fbe35b4e\") " pod="opendatahub/opendatahub-operator-controller-manager-66b5c4b596-9klsp" Jul 20 21:11:07.508612 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:07.508498 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1a9af75b-a8d9-47b3-976a-be47fbe35b4e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-66b5c4b596-9klsp\" (UID: \"1a9af75b-a8d9-47b3-976a-be47fbe35b4e\") " pod="opendatahub/opendatahub-operator-controller-manager-66b5c4b596-9klsp" Jul 20 21:11:07.508612 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:07.508540 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1a9af75b-a8d9-47b3-976a-be47fbe35b4e-webhook-cert\") pod \"opendatahub-operator-controller-manager-66b5c4b596-9klsp\" (UID: \"1a9af75b-a8d9-47b3-976a-be47fbe35b4e\") " pod="opendatahub/opendatahub-operator-controller-manager-66b5c4b596-9klsp" Jul 20 21:11:07.508776 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:07.508626 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1a9af75b-a8d9-47b3-976a-be47fbe35b4e-manifests\") pod \"opendatahub-operator-controller-manager-66b5c4b596-9klsp\" (UID: \"1a9af75b-a8d9-47b3-976a-be47fbe35b4e\") " pod="opendatahub/opendatahub-operator-controller-manager-66b5c4b596-9klsp" Jul 20 21:11:07.522698 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:07.522659 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:11:07.610717 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:07.610113 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9kct\" (UniqueName: \"kubernetes.io/projected/1a9af75b-a8d9-47b3-976a-be47fbe35b4e-kube-api-access-b9kct\") pod \"opendatahub-operator-controller-manager-66b5c4b596-9klsp\" (UID: \"1a9af75b-a8d9-47b3-976a-be47fbe35b4e\") " pod="opendatahub/opendatahub-operator-controller-manager-66b5c4b596-9klsp" Jul 20 21:11:07.610717 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:07.610212 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1a9af75b-a8d9-47b3-976a-be47fbe35b4e-tmp\") pod \"opendatahub-operator-controller-manager-66b5c4b596-9klsp\" (UID: \"1a9af75b-a8d9-47b3-976a-be47fbe35b4e\") " pod="opendatahub/opendatahub-operator-controller-manager-66b5c4b596-9klsp" Jul 20 21:11:07.610717 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:07.610251 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1a9af75b-a8d9-47b3-976a-be47fbe35b4e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-66b5c4b596-9klsp\" (UID: \"1a9af75b-a8d9-47b3-976a-be47fbe35b4e\") " pod="opendatahub/opendatahub-operator-controller-manager-66b5c4b596-9klsp" Jul 20 21:11:07.610717 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:07.610277 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1a9af75b-a8d9-47b3-976a-be47fbe35b4e-webhook-cert\") pod \"opendatahub-operator-controller-manager-66b5c4b596-9klsp\" (UID: \"1a9af75b-a8d9-47b3-976a-be47fbe35b4e\") " pod="opendatahub/opendatahub-operator-controller-manager-66b5c4b596-9klsp" Jul 20 21:11:07.610717 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:07.610338 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1a9af75b-a8d9-47b3-976a-be47fbe35b4e-manifests\") pod \"opendatahub-operator-controller-manager-66b5c4b596-9klsp\" (UID: \"1a9af75b-a8d9-47b3-976a-be47fbe35b4e\") " pod="opendatahub/opendatahub-operator-controller-manager-66b5c4b596-9klsp" Jul 20 21:11:07.611116 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:07.610765 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1a9af75b-a8d9-47b3-976a-be47fbe35b4e-manifests\") pod \"opendatahub-operator-controller-manager-66b5c4b596-9klsp\" (UID: \"1a9af75b-a8d9-47b3-976a-be47fbe35b4e\") " pod="opendatahub/opendatahub-operator-controller-manager-66b5c4b596-9klsp" Jul 20 21:11:07.613616 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:07.613572 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1a9af75b-a8d9-47b3-976a-be47fbe35b4e-tmp\") pod \"opendatahub-operator-controller-manager-66b5c4b596-9klsp\" (UID: \"1a9af75b-a8d9-47b3-976a-be47fbe35b4e\") " pod="opendatahub/opendatahub-operator-controller-manager-66b5c4b596-9klsp" Jul 20 21:11:07.613867 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:07.613845 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1a9af75b-a8d9-47b3-976a-be47fbe35b4e-webhook-cert\") pod \"opendatahub-operator-controller-manager-66b5c4b596-9klsp\" (UID: \"1a9af75b-a8d9-47b3-976a-be47fbe35b4e\") " pod="opendatahub/opendatahub-operator-controller-manager-66b5c4b596-9klsp" Jul 20 21:11:07.616013 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:07.615972 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1a9af75b-a8d9-47b3-976a-be47fbe35b4e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-66b5c4b596-9klsp\" (UID: \"1a9af75b-a8d9-47b3-976a-be47fbe35b4e\") " pod="opendatahub/opendatahub-operator-controller-manager-66b5c4b596-9klsp" Jul 20 21:11:07.622463 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:07.622406 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9kct\" (UniqueName: \"kubernetes.io/projected/1a9af75b-a8d9-47b3-976a-be47fbe35b4e-kube-api-access-b9kct\") pod \"opendatahub-operator-controller-manager-66b5c4b596-9klsp\" (UID: \"1a9af75b-a8d9-47b3-976a-be47fbe35b4e\") " pod="opendatahub/opendatahub-operator-controller-manager-66b5c4b596-9klsp" Jul 20 21:11:07.640817 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:07.640778 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:11:07.672823 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:07.672704 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-66b5c4b596-9klsp" Jul 20 21:11:07.737545 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:07.736827 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:11:07.839265 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:07.839225 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:11:07.858769 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:07.858500 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-66b5c4b596-9klsp"] Jul 20 21:11:07.864586 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:11:07.864543 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a9af75b_a8d9_47b3_976a_be47fbe35b4e.slice/crio-1344e6cfe3682bab99072b192ddd994c4f829c020242359bd5940b191d9c139b WatchSource:0}: Error finding container 1344e6cfe3682bab99072b192ddd994c4f829c020242359bd5940b191d9c139b: Status 404 returned error can't find the container with id 1344e6cfe3682bab99072b192ddd994c4f829c020242359bd5940b191d9c139b Jul 20 21:11:07.942396 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:07.942363 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:11:08.053308 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:08.053257 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:11:08.088341 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:08.088292 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-66b5c4b596-9klsp" event={"ID":"1a9af75b-a8d9-47b3-976a-be47fbe35b4e","Type":"ContainerStarted","Data":"1344e6cfe3682bab99072b192ddd994c4f829c020242359bd5940b191d9c139b"} Jul 20 21:11:10.101919 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:10.101872 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7475cd49b4-ns279" event={"ID":"b047db61-e3ec-4c54-83f9-d14bc3dd38f5","Type":"ContainerStarted","Data":"46ffa9cac2c7465289f2fcaf152b177c259a5d4eae249244978c319d7258ff7a"} Jul 20 21:11:10.102377 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:10.101961 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7475cd49b4-ns279" Jul 20 21:11:10.131471 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:10.131420 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7475cd49b4-ns279" podStartSLOduration=1.8278836840000001 podStartE2EDuration="4.131399151s" totalImagesPullingTime="2.303515467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 21:11:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:11:10.130913853 +0000 UTC m=+509.456616798" watchObservedRunningTime="2026-07-20 21:11:10.131399151 +0000 UTC m=+509.457102094" Jul 20 21:11:12.112558 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:12.112523 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-66b5c4b596-9klsp" event={"ID":"1a9af75b-a8d9-47b3-976a-be47fbe35b4e","Type":"ContainerStarted","Data":"4802d7b1c38fe5217c5cec17e8b4e834ba87fc744091e5431794bb5d5c167662"} Jul 20 21:11:13.118762 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:13.118721 2562 generic.go:356] "Generic (PLEG): container finished" podID="1a9af75b-a8d9-47b3-976a-be47fbe35b4e" containerID="4802d7b1c38fe5217c5cec17e8b4e834ba87fc744091e5431794bb5d5c167662" exitCode=0 Jul 20 21:11:13.119192 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:13.118763 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-66b5c4b596-9klsp" event={"ID":"1a9af75b-a8d9-47b3-976a-be47fbe35b4e","Type":"ContainerDied","Data":"4802d7b1c38fe5217c5cec17e8b4e834ba87fc744091e5431794bb5d5c167662"} Jul 20 21:11:14.125772 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:14.125735 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-66b5c4b596-9klsp" event={"ID":"1a9af75b-a8d9-47b3-976a-be47fbe35b4e","Type":"ContainerStarted","Data":"9fc74a0c736e5d27fb4f17b58f35631b8ed889cbe7dff1079948ef0486495fa4"} Jul 20 21:11:14.126214 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:14.125853 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-66b5c4b596-9klsp" Jul 20 21:11:14.152967 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:14.152904 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-66b5c4b596-9klsp" podStartSLOduration=2.185766529 podStartE2EDuration="7.152880917s" totalImagesPullingTime="3.902162317s" totalInitContainerRuntime="1.064952071s" isStatefulPod=false podCreationTimestamp="2026-07-20 21:11:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:11:14.151232395 +0000 UTC m=+513.476935340" watchObservedRunningTime="2026-07-20 21:11:14.152880917 +0000 UTC m=+513.478583861" Jul 20 21:11:21.109277 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:21.109242 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7475cd49b4-ns279" Jul 20 21:11:25.133225 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:25.133195 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-66b5c4b596-9klsp" Jul 20 21:11:26.916542 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:26.916496 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:11:27.081933 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:27.081893 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:11:27.492634 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:27.492597 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:11:27.631363 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:27.631330 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5cfcf6b877-tnp27"] Jul 20 21:11:27.726048 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:27.726009 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5cfcf6b877-tnp27"] Jul 20 21:11:27.726241 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:27.726145 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5cfcf6b877-tnp27" Jul 20 21:11:27.730643 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:27.730621 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 20 21:11:27.730900 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:27.730878 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 20 21:11:27.731039 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:27.730906 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-fmzrc\"" Jul 20 21:11:27.731039 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:27.730964 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 20 21:11:27.731667 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:27.731647 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 20 21:11:27.803675 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:27.803638 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bba992c-dc11-4a0e-b5bd-0a2e631bf26c-tls-certs\") pod \"kube-auth-proxy-5cfcf6b877-tnp27\" (UID: \"0bba992c-dc11-4a0e-b5bd-0a2e631bf26c\") " pod="openshift-ingress/kube-auth-proxy-5cfcf6b877-tnp27" Jul 20 21:11:27.803875 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:27.803710 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0bba992c-dc11-4a0e-b5bd-0a2e631bf26c-tmp\") pod \"kube-auth-proxy-5cfcf6b877-tnp27\" (UID: \"0bba992c-dc11-4a0e-b5bd-0a2e631bf26c\") " pod="openshift-ingress/kube-auth-proxy-5cfcf6b877-tnp27" Jul 20 21:11:27.803875 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:27.803740 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4wqg\" (UniqueName: \"kubernetes.io/projected/0bba992c-dc11-4a0e-b5bd-0a2e631bf26c-kube-api-access-j4wqg\") pod \"kube-auth-proxy-5cfcf6b877-tnp27\" (UID: \"0bba992c-dc11-4a0e-b5bd-0a2e631bf26c\") " pod="openshift-ingress/kube-auth-proxy-5cfcf6b877-tnp27" Jul 20 21:11:27.904797 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:27.904758 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0bba992c-dc11-4a0e-b5bd-0a2e631bf26c-tmp\") pod \"kube-auth-proxy-5cfcf6b877-tnp27\" (UID: \"0bba992c-dc11-4a0e-b5bd-0a2e631bf26c\") " pod="openshift-ingress/kube-auth-proxy-5cfcf6b877-tnp27" Jul 20 21:11:27.904987 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:27.904816 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4wqg\" (UniqueName: \"kubernetes.io/projected/0bba992c-dc11-4a0e-b5bd-0a2e631bf26c-kube-api-access-j4wqg\") pod \"kube-auth-proxy-5cfcf6b877-tnp27\" (UID: \"0bba992c-dc11-4a0e-b5bd-0a2e631bf26c\") " pod="openshift-ingress/kube-auth-proxy-5cfcf6b877-tnp27" Jul 20 21:11:27.904987 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:27.904936 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bba992c-dc11-4a0e-b5bd-0a2e631bf26c-tls-certs\") pod \"kube-auth-proxy-5cfcf6b877-tnp27\" (UID: \"0bba992c-dc11-4a0e-b5bd-0a2e631bf26c\") " pod="openshift-ingress/kube-auth-proxy-5cfcf6b877-tnp27" Jul 20 21:11:27.905100 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:11:27.905082 2562 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 20 21:11:27.905204 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:11:27.905189 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bba992c-dc11-4a0e-b5bd-0a2e631bf26c-tls-certs podName:0bba992c-dc11-4a0e-b5bd-0a2e631bf26c nodeName:}" failed. No retries permitted until 2026-07-20 21:11:28.405166805 +0000 UTC m=+527.730869729 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/0bba992c-dc11-4a0e-b5bd-0a2e631bf26c-tls-certs") pod "kube-auth-proxy-5cfcf6b877-tnp27" (UID: "0bba992c-dc11-4a0e-b5bd-0a2e631bf26c") : secret "kube-auth-proxy-tls" not found Jul 20 21:11:27.907538 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:27.907500 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0bba992c-dc11-4a0e-b5bd-0a2e631bf26c-tmp\") pod \"kube-auth-proxy-5cfcf6b877-tnp27\" (UID: \"0bba992c-dc11-4a0e-b5bd-0a2e631bf26c\") " pod="openshift-ingress/kube-auth-proxy-5cfcf6b877-tnp27" Jul 20 21:11:27.945972 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:27.945943 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4wqg\" (UniqueName: \"kubernetes.io/projected/0bba992c-dc11-4a0e-b5bd-0a2e631bf26c-kube-api-access-j4wqg\") pod \"kube-auth-proxy-5cfcf6b877-tnp27\" (UID: \"0bba992c-dc11-4a0e-b5bd-0a2e631bf26c\") " pod="openshift-ingress/kube-auth-proxy-5cfcf6b877-tnp27" Jul 20 21:11:27.955404 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:27.955367 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:11:28.118490 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:28.118392 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:11:28.240849 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:28.240821 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:11:28.331700 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:28.331667 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:11:28.410246 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:28.410150 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bba992c-dc11-4a0e-b5bd-0a2e631bf26c-tls-certs\") pod \"kube-auth-proxy-5cfcf6b877-tnp27\" (UID: \"0bba992c-dc11-4a0e-b5bd-0a2e631bf26c\") " pod="openshift-ingress/kube-auth-proxy-5cfcf6b877-tnp27" Jul 20 21:11:28.412584 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:28.412555 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bba992c-dc11-4a0e-b5bd-0a2e631bf26c-tls-certs\") pod \"kube-auth-proxy-5cfcf6b877-tnp27\" (UID: \"0bba992c-dc11-4a0e-b5bd-0a2e631bf26c\") " pod="openshift-ingress/kube-auth-proxy-5cfcf6b877-tnp27" Jul 20 21:11:28.636810 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:28.636775 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5cfcf6b877-tnp27" Jul 20 21:11:28.858083 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:28.858054 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5cfcf6b877-tnp27"] Jul 20 21:11:28.860481 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:11:28.860457 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bba992c_dc11_4a0e_b5bd_0a2e631bf26c.slice/crio-8b376a6269c0baced7cb70832be0eb62eaaca09092f43f165bfebe1e449be570 WatchSource:0}: Error finding container 8b376a6269c0baced7cb70832be0eb62eaaca09092f43f165bfebe1e449be570: Status 404 returned error can't find the container with id 8b376a6269c0baced7cb70832be0eb62eaaca09092f43f165bfebe1e449be570 Jul 20 21:11:28.937927 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:28.937893 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:11:29.194905 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:29.194873 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5cfcf6b877-tnp27" event={"ID":"0bba992c-dc11-4a0e-b5bd-0a2e631bf26c","Type":"ContainerStarted","Data":"8b376a6269c0baced7cb70832be0eb62eaaca09092f43f165bfebe1e449be570"} Jul 20 21:11:29.220215 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:29.220184 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:11:29.514027 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:29.513956 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:11:29.597777 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:29.597742 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:11:30.613820 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:30.613789 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:11:32.210044 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:32.210009 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5cfcf6b877-tnp27" event={"ID":"0bba992c-dc11-4a0e-b5bd-0a2e631bf26c","Type":"ContainerStarted","Data":"76634feb17547bff67771f8274d9759ce248d87944dada0195a82a7cb8551047"} Jul 20 21:11:32.230907 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:32.230855 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5cfcf6b877-tnp27" podStartSLOduration=2.485901381 podStartE2EDuration="5.230840451s" totalImagesPullingTime="2.74493907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 21:11:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:11:32.230199417 +0000 UTC m=+531.555902360" watchObservedRunningTime="2026-07-20 21:11:32.230840451 +0000 UTC m=+531.556543393" Jul 20 21:11:35.457055 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:35.456966 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:11:35.595717 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:35.595683 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:11:36.626427 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:36.626393 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:11:36.849310 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:36.849253 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:11:36.989471 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:36.989387 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:11:39.856660 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:39.856628 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:11:40.064615 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:40.064586 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:11:40.246773 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:40.246690 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:11:40.426789 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:40.426744 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:11:41.032915 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:41.032879 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:11:41.289424 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:41.289340 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:11:41.385845 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:41.385808 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:11:41.490549 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:41.490520 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:11:42.263977 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:42.263942 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:11:42.338830 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:42.338795 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:11:44.185245 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:44.185212 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:11:44.434952 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:44.434917 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:11:56.431252 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:56.431210 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:11:56.577128 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:11:56.577090 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:12:19.278278 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:12:19.278244 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:12:19.345165 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:12:19.345115 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:12:28.603592 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:12:28.603560 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:12:29.700538 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:12:29.700498 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:12:29.753707 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:12:29.753671 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:12:29.898501 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:12:29.898467 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:12:34.065530 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:12:34.065497 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:12:34.222458 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:12:34.222422 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:12:53.668221 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:12:53.668183 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6bd98d7844-x6slf"] Jul 20 21:12:53.670620 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:12:53.670597 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bd98d7844-x6slf" Jul 20 21:12:53.673083 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:12:53.673053 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 20 21:12:53.673259 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:12:53.673200 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 20 21:12:53.673342 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:12:53.673287 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 20 21:12:53.673342 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:12:53.673299 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 20 21:12:53.674329 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:12:53.674301 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 20 21:12:53.674417 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:12:53.674339 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 20 21:12:53.674590 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:12:53.674568 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-98t82\"" Jul 20 21:12:53.674667 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:12:53.674572 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 20 21:12:53.681253 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:12:53.681231 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 20 21:12:53.682809 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:12:53.682783 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bd98d7844-x6slf"] Jul 20 21:12:53.722606 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:12:53.722571 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:12:53.767366 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:12:53.767330 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f498ba96-6fee-4314-a284-af05c8147096-oauth-serving-cert\") pod \"console-6bd98d7844-x6slf\" (UID: \"f498ba96-6fee-4314-a284-af05c8147096\") " pod="openshift-console/console-6bd98d7844-x6slf" Jul 20 21:12:53.767519 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:12:53.767381 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mgmv\" (UniqueName: \"kubernetes.io/projected/f498ba96-6fee-4314-a284-af05c8147096-kube-api-access-2mgmv\") pod \"console-6bd98d7844-x6slf\" (UID: \"f498ba96-6fee-4314-a284-af05c8147096\") " pod="openshift-console/console-6bd98d7844-x6slf" Jul 20 21:12:53.767519 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:12:53.767420 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f498ba96-6fee-4314-a284-af05c8147096-service-ca\") pod \"console-6bd98d7844-x6slf\" (UID: \"f498ba96-6fee-4314-a284-af05c8147096\") " pod="openshift-console/console-6bd98d7844-x6slf" Jul 20 21:12:53.767519 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:12:53.767461 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f498ba96-6fee-4314-a284-af05c8147096-console-serving-cert\") pod \"console-6bd98d7844-x6slf\" (UID: \"f498ba96-6fee-4314-a284-af05c8147096\") " pod="openshift-console/console-6bd98d7844-x6slf" Jul 20 21:12:53.767519 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:12:53.767476 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f498ba96-6fee-4314-a284-af05c8147096-console-config\") pod \"console-6bd98d7844-x6slf\" (UID: \"f498ba96-6fee-4314-a284-af05c8147096\") " pod="openshift-console/console-6bd98d7844-x6slf" Jul 20 21:12:53.767697 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:12:53.767520 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f498ba96-6fee-4314-a284-af05c8147096-trusted-ca-bundle\") pod \"console-6bd98d7844-x6slf\" (UID: \"f498ba96-6fee-4314-a284-af05c8147096\") " pod="openshift-console/console-6bd98d7844-x6slf" Jul 20 21:12:53.767697 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:12:53.767591 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f498ba96-6fee-4314-a284-af05c8147096-console-oauth-config\") pod \"console-6bd98d7844-x6slf\" (UID: \"f498ba96-6fee-4314-a284-af05c8147096\") " pod="openshift-console/console-6bd98d7844-x6slf" Jul 20 21:12:53.822829 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:12:53.822798 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:12:53.868815 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:12:53.868784 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f498ba96-6fee-4314-a284-af05c8147096-console-oauth-config\") pod \"console-6bd98d7844-x6slf\" (UID: \"f498ba96-6fee-4314-a284-af05c8147096\") " pod="openshift-console/console-6bd98d7844-x6slf" Jul 20 21:12:53.868939 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:12:53.868827 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f498ba96-6fee-4314-a284-af05c8147096-oauth-serving-cert\") pod \"console-6bd98d7844-x6slf\" (UID: \"f498ba96-6fee-4314-a284-af05c8147096\") " pod="openshift-console/console-6bd98d7844-x6slf" Jul 20 21:12:53.868982 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:12:53.868951 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mgmv\" (UniqueName: \"kubernetes.io/projected/f498ba96-6fee-4314-a284-af05c8147096-kube-api-access-2mgmv\") pod \"console-6bd98d7844-x6slf\" (UID: \"f498ba96-6fee-4314-a284-af05c8147096\") " pod="openshift-console/console-6bd98d7844-x6slf" Jul 20 21:12:53.869015 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:12:53.868984 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f498ba96-6fee-4314-a284-af05c8147096-service-ca\") pod \"console-6bd98d7844-x6slf\" (UID: \"f498ba96-6fee-4314-a284-af05c8147096\") " pod="openshift-console/console-6bd98d7844-x6slf" Jul 20 21:12:53.869073 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:12:53.869056 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f498ba96-6fee-4314-a284-af05c8147096-console-serving-cert\") pod \"console-6bd98d7844-x6slf\" (UID: \"f498ba96-6fee-4314-a284-af05c8147096\") " pod="openshift-console/console-6bd98d7844-x6slf" Jul 20 21:12:53.869119 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:12:53.869087 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f498ba96-6fee-4314-a284-af05c8147096-console-config\") pod \"console-6bd98d7844-x6slf\" (UID: \"f498ba96-6fee-4314-a284-af05c8147096\") " pod="openshift-console/console-6bd98d7844-x6slf" Jul 20 21:12:53.869119 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:12:53.869112 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f498ba96-6fee-4314-a284-af05c8147096-trusted-ca-bundle\") pod \"console-6bd98d7844-x6slf\" (UID: \"f498ba96-6fee-4314-a284-af05c8147096\") " pod="openshift-console/console-6bd98d7844-x6slf" Jul 20 21:12:53.869674 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:12:53.869644 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f498ba96-6fee-4314-a284-af05c8147096-oauth-serving-cert\") pod \"console-6bd98d7844-x6slf\" (UID: \"f498ba96-6fee-4314-a284-af05c8147096\") " pod="openshift-console/console-6bd98d7844-x6slf" Jul 20 21:12:53.869790 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:12:53.869765 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f498ba96-6fee-4314-a284-af05c8147096-service-ca\") pod \"console-6bd98d7844-x6slf\" (UID: \"f498ba96-6fee-4314-a284-af05c8147096\") " pod="openshift-console/console-6bd98d7844-x6slf" Jul 20 21:12:53.869856 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:12:53.869790 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f498ba96-6fee-4314-a284-af05c8147096-console-config\") pod \"console-6bd98d7844-x6slf\" (UID: \"f498ba96-6fee-4314-a284-af05c8147096\") " pod="openshift-console/console-6bd98d7844-x6slf" Jul 20 21:12:53.869951 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:12:53.869932 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f498ba96-6fee-4314-a284-af05c8147096-trusted-ca-bundle\") pod \"console-6bd98d7844-x6slf\" (UID: \"f498ba96-6fee-4314-a284-af05c8147096\") " pod="openshift-console/console-6bd98d7844-x6slf" Jul 20 21:12:53.871388 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:12:53.871354 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f498ba96-6fee-4314-a284-af05c8147096-console-oauth-config\") pod \"console-6bd98d7844-x6slf\" (UID: \"f498ba96-6fee-4314-a284-af05c8147096\") " pod="openshift-console/console-6bd98d7844-x6slf" Jul 20 21:12:53.871507 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:12:53.871467 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f498ba96-6fee-4314-a284-af05c8147096-console-serving-cert\") pod \"console-6bd98d7844-x6slf\" (UID: \"f498ba96-6fee-4314-a284-af05c8147096\") " pod="openshift-console/console-6bd98d7844-x6slf" Jul 20 21:12:53.878722 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:12:53.878694 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mgmv\" (UniqueName: \"kubernetes.io/projected/f498ba96-6fee-4314-a284-af05c8147096-kube-api-access-2mgmv\") pod \"console-6bd98d7844-x6slf\" (UID: \"f498ba96-6fee-4314-a284-af05c8147096\") " pod="openshift-console/console-6bd98d7844-x6slf" Jul 20 21:12:53.980795 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:12:53.980702 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bd98d7844-x6slf" Jul 20 21:12:54.118149 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:12:54.118100 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bd98d7844-x6slf"] Jul 20 21:12:54.121733 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:12:54.121703 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf498ba96_6fee_4314_a284_af05c8147096.slice/crio-d46d67e719af24a82d3b75af80f48a84367c51b77e4679ed1a56fab5295f91e3 WatchSource:0}: Error finding container d46d67e719af24a82d3b75af80f48a84367c51b77e4679ed1a56fab5295f91e3: Status 404 returned error can't find the container with id d46d67e719af24a82d3b75af80f48a84367c51b77e4679ed1a56fab5295f91e3 Jul 20 21:12:54.587538 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:12:54.587491 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bd98d7844-x6slf" event={"ID":"f498ba96-6fee-4314-a284-af05c8147096","Type":"ContainerStarted","Data":"79d8af383333d007295f84abe7fb9659f05c44dcf72a365eaddcd7fc7a290c21"} Jul 20 21:12:54.587764 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:12:54.587555 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bd98d7844-x6slf" event={"ID":"f498ba96-6fee-4314-a284-af05c8147096","Type":"ContainerStarted","Data":"d46d67e719af24a82d3b75af80f48a84367c51b77e4679ed1a56fab5295f91e3"} Jul 20 21:12:54.612437 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:12:54.612382 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6bd98d7844-x6slf" podStartSLOduration=1.612366077 podStartE2EDuration="1.612366077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 21:12:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:12:54.611653079 +0000 UTC m=+613.937356024" watchObservedRunningTime="2026-07-20 21:12:54.612366077 +0000 UTC m=+613.938069093" Jul 20 21:12:59.930310 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:12:59.930272 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:13:00.537630 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:00.537592 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:13:00.955245 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:00.955128 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:13:01.056881 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:01.056847 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:13:03.981396 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:03.981308 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6bd98d7844-x6slf" Jul 20 21:13:03.981812 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:03.981600 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6bd98d7844-x6slf" Jul 20 21:13:03.986233 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:03.986210 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6bd98d7844-x6slf" Jul 20 21:13:04.642257 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:04.642226 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6bd98d7844-x6slf" Jul 20 21:13:04.866446 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:04.866408 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:13:04.974161 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:04.974056 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:13:05.085414 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:05.085380 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:13:05.220311 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:05.220283 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:13:05.822357 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:05.822327 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:13:08.756424 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:08.756386 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:13:13.763432 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:13.763390 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:13:18.356013 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:18.355980 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:13:18.451569 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:18.451526 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:13:19.031937 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:19.031905 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:13:19.130045 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:19.130011 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:13:22.431738 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:22.431702 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:13:22.735560 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:22.735471 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:13:28.972654 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:28.972617 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:13:29.861754 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:29.861717 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:13:31.631501 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:31.631469 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:13:33.627785 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:33.627744 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hhnpn"] Jul 20 21:13:33.631814 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:33.631792 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hhnpn" Jul 20 21:13:33.634897 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:33.634871 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 20 21:13:33.636588 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:33.636338 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 20 21:13:33.636588 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:33.636515 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-fmgz5\"" Jul 20 21:13:33.651466 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:33.651441 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hhnpn"] Jul 20 21:13:33.729631 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:33.729598 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz9sv\" (UniqueName: \"kubernetes.io/projected/472af867-3e66-4dcf-a643-400f4a63a64f-kube-api-access-fz9sv\") pod \"limitador-operator-controller-manager-859d7b78bd-hhnpn\" (UID: \"472af867-3e66-4dcf-a643-400f4a63a64f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hhnpn" Jul 20 21:13:33.830992 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:33.830956 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz9sv\" (UniqueName: \"kubernetes.io/projected/472af867-3e66-4dcf-a643-400f4a63a64f-kube-api-access-fz9sv\") pod \"limitador-operator-controller-manager-859d7b78bd-hhnpn\" (UID: \"472af867-3e66-4dcf-a643-400f4a63a64f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hhnpn" Jul 20 21:13:33.841285 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:33.841258 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz9sv\" (UniqueName: \"kubernetes.io/projected/472af867-3e66-4dcf-a643-400f4a63a64f-kube-api-access-fz9sv\") pod \"limitador-operator-controller-manager-859d7b78bd-hhnpn\" (UID: \"472af867-3e66-4dcf-a643-400f4a63a64f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hhnpn" Jul 20 21:13:33.951219 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:33.951119 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hhnpn" Jul 20 21:13:34.083479 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:34.083444 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hhnpn"] Jul 20 21:13:34.086812 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:13:34.086778 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod472af867_3e66_4dcf_a643_400f4a63a64f.slice/crio-6ad041340dc9eeb1775469985bd21078c3392bc35c928fc18721453b522231ee WatchSource:0}: Error finding container 6ad041340dc9eeb1775469985bd21078c3392bc35c928fc18721453b522231ee: Status 404 returned error can't find the container with id 6ad041340dc9eeb1775469985bd21078c3392bc35c928fc18721453b522231ee Jul 20 21:13:34.195867 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:34.195830 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:13:34.292048 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:34.292013 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:13:34.784024 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:34.783992 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hhnpn" event={"ID":"472af867-3e66-4dcf-a643-400f4a63a64f","Type":"ContainerStarted","Data":"6ad041340dc9eeb1775469985bd21078c3392bc35c928fc18721453b522231ee"} Jul 20 21:13:36.799381 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:36.799338 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hhnpn" event={"ID":"472af867-3e66-4dcf-a643-400f4a63a64f","Type":"ContainerStarted","Data":"7a0cb09af5918356eb34a7cd40c03102316bea6e2546f68eb33eac0f80f3bb53"} Jul 20 21:13:36.799754 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:36.799421 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hhnpn" Jul 20 21:13:36.839437 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:36.839376 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hhnpn" podStartSLOduration=1.8459511979999998 podStartE2EDuration="3.839358855s" totalImagesPullingTime="1.993407657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 21:13:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:13:36.837795612 +0000 UTC m=+656.163498556" watchObservedRunningTime="2026-07-20 21:13:36.839358855 +0000 UTC m=+656.165061797" Jul 20 21:13:39.534279 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:39.534238 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:13:39.602162 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:39.602078 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:13:40.143031 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:40.142998 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:13:40.244493 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:40.244463 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:13:43.591466 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:43.591436 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:13:43.683941 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:43.683898 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:13:46.157283 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:46.157246 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:13:46.313761 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:46.313728 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:13:47.806181 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:47.806125 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hhnpn" Jul 20 21:13:52.601917 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:52.601880 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:13:52.671742 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:52.671707 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:13:58.375036 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:58.375002 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:13:58.487044 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:58.487016 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:13:58.981412 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:58.981380 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:13:59.081078 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:59.081048 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:13:59.221720 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:13:59.221684 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:14:02.773907 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:14:02.773875 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:14:02.904346 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:14:02.904316 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:14:06.184803 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:14:06.184753 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:14:06.309456 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:14:06.309420 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:14:13.267933 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:14:13.267903 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:14:13.729764 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:14:13.729679 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:14:13.830787 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:14:13.830761 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:14:14.355705 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:14:14.355672 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:14:14.453405 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:14:14.453372 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:14:29.806742 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:14:29.806703 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:14:29.903999 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:14:29.903963 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:14:29.980688 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:14:29.980652 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:14:30.657042 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:14:30.657008 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:14:30.718665 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:14:30.718629 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:14:30.811163 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:14:30.811114 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-ftlx6"] Jul 20 21:14:30.816806 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:14:30.816784 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-ftlx6" Jul 20 21:14:30.819702 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:14:30.819681 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-kn45b\"" Jul 20 21:14:30.821933 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:14:30.821910 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-ftlx6"] Jul 20 21:14:30.839677 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:14:30.839650 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn6wp\" (UniqueName: \"kubernetes.io/projected/2ffa3370-e4b4-45e6-86c2-23db0ea0b21d-kube-api-access-hn6wp\") pod \"authorino-75d556bd77-ftlx6\" (UID: \"2ffa3370-e4b4-45e6-86c2-23db0ea0b21d\") " pod="kuadrant-system/authorino-75d556bd77-ftlx6" Jul 20 21:14:30.940991 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:14:30.940908 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn6wp\" (UniqueName: \"kubernetes.io/projected/2ffa3370-e4b4-45e6-86c2-23db0ea0b21d-kube-api-access-hn6wp\") pod \"authorino-75d556bd77-ftlx6\" (UID: \"2ffa3370-e4b4-45e6-86c2-23db0ea0b21d\") " pod="kuadrant-system/authorino-75d556bd77-ftlx6" Jul 20 21:14:30.949241 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:14:30.949210 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn6wp\" (UniqueName: \"kubernetes.io/projected/2ffa3370-e4b4-45e6-86c2-23db0ea0b21d-kube-api-access-hn6wp\") pod \"authorino-75d556bd77-ftlx6\" (UID: \"2ffa3370-e4b4-45e6-86c2-23db0ea0b21d\") " pod="kuadrant-system/authorino-75d556bd77-ftlx6" Jul 20 21:14:31.127126 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:14:31.127093 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-ftlx6" Jul 20 21:14:31.270681 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:14:31.270645 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ffa3370_e4b4_45e6_86c2_23db0ea0b21d.slice/crio-d0384702126ab7f1ad7ec33ac9078f848e8e6ae3d0bb5c55a3a98d1a32c4b9f6 WatchSource:0}: Error finding container d0384702126ab7f1ad7ec33ac9078f848e8e6ae3d0bb5c55a3a98d1a32c4b9f6: Status 404 returned error can't find the container with id d0384702126ab7f1ad7ec33ac9078f848e8e6ae3d0bb5c55a3a98d1a32c4b9f6 Jul 20 21:14:31.271901 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:14:31.271882 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 21:14:31.273011 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:14:31.272987 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-ftlx6"] Jul 20 21:14:32.056623 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:14:32.056489 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-ftlx6" event={"ID":"2ffa3370-e4b4-45e6-86c2-23db0ea0b21d","Type":"ContainerStarted","Data":"d0384702126ab7f1ad7ec33ac9078f848e8e6ae3d0bb5c55a3a98d1a32c4b9f6"} Jul 20 21:14:35.073000 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:14:35.072960 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-ftlx6" event={"ID":"2ffa3370-e4b4-45e6-86c2-23db0ea0b21d","Type":"ContainerStarted","Data":"9efbd73a19a49b5ede688abf67ddd77e28fc3a9f8642a2b8dc55ab001faf1a8a"} Jul 20 21:14:35.089567 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:14:35.089518 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-ftlx6" podStartSLOduration=1.668212151 podStartE2EDuration="5.089502065s" totalImagesPullingTime="3.421289914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 21:14:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:14:35.088647034 +0000 UTC m=+714.414349977" watchObservedRunningTime="2026-07-20 21:14:35.089502065 +0000 UTC m=+714.415205007" Jul 20 21:14:49.703415 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:14:49.703381 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:14:49.801833 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:14:49.801801 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:14:50.360455 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:14:50.360423 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:15:02.270495 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:02.270457 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:15:02.472415 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:02.472387 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:15:03.916192 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:03.916153 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-ftlx6"] Jul 20 21:15:03.916570 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:03.916367 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-ftlx6" podUID="2ffa3370-e4b4-45e6-86c2-23db0ea0b21d" containerName="authorino" containerID="cri-o://9efbd73a19a49b5ede688abf67ddd77e28fc3a9f8642a2b8dc55ab001faf1a8a" gracePeriod=30 Jul 20 21:15:04.162398 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:04.162374 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-ftlx6" Jul 20 21:15:04.212047 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:04.211963 2562 generic.go:356] "Generic (PLEG): container finished" podID="2ffa3370-e4b4-45e6-86c2-23db0ea0b21d" containerID="9efbd73a19a49b5ede688abf67ddd77e28fc3a9f8642a2b8dc55ab001faf1a8a" exitCode=0 Jul 20 21:15:04.212047 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:04.212002 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-ftlx6" Jul 20 21:15:04.212249 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:04.212057 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-ftlx6" event={"ID":"2ffa3370-e4b4-45e6-86c2-23db0ea0b21d","Type":"ContainerDied","Data":"9efbd73a19a49b5ede688abf67ddd77e28fc3a9f8642a2b8dc55ab001faf1a8a"} Jul 20 21:15:04.212249 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:04.212098 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-ftlx6" event={"ID":"2ffa3370-e4b4-45e6-86c2-23db0ea0b21d","Type":"ContainerDied","Data":"d0384702126ab7f1ad7ec33ac9078f848e8e6ae3d0bb5c55a3a98d1a32c4b9f6"} Jul 20 21:15:04.212249 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:04.212115 2562 scope.go:117] "RemoveContainer" containerID="9efbd73a19a49b5ede688abf67ddd77e28fc3a9f8642a2b8dc55ab001faf1a8a" Jul 20 21:15:04.220367 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:04.220349 2562 scope.go:117] "RemoveContainer" containerID="9efbd73a19a49b5ede688abf67ddd77e28fc3a9f8642a2b8dc55ab001faf1a8a" Jul 20 21:15:04.220631 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:15:04.220610 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9efbd73a19a49b5ede688abf67ddd77e28fc3a9f8642a2b8dc55ab001faf1a8a\": container with ID starting with 9efbd73a19a49b5ede688abf67ddd77e28fc3a9f8642a2b8dc55ab001faf1a8a not found: ID does not exist" containerID="9efbd73a19a49b5ede688abf67ddd77e28fc3a9f8642a2b8dc55ab001faf1a8a" Jul 20 21:15:04.220702 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:04.220645 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9efbd73a19a49b5ede688abf67ddd77e28fc3a9f8642a2b8dc55ab001faf1a8a"} err="failed to get container status \"9efbd73a19a49b5ede688abf67ddd77e28fc3a9f8642a2b8dc55ab001faf1a8a\": rpc error: code = NotFound desc = could not find container \"9efbd73a19a49b5ede688abf67ddd77e28fc3a9f8642a2b8dc55ab001faf1a8a\": container with ID starting with 9efbd73a19a49b5ede688abf67ddd77e28fc3a9f8642a2b8dc55ab001faf1a8a not found: ID does not exist" Jul 20 21:15:04.240961 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:04.240939 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hn6wp\" (UniqueName: \"kubernetes.io/projected/2ffa3370-e4b4-45e6-86c2-23db0ea0b21d-kube-api-access-hn6wp\") pod \"2ffa3370-e4b4-45e6-86c2-23db0ea0b21d\" (UID: \"2ffa3370-e4b4-45e6-86c2-23db0ea0b21d\") " Jul 20 21:15:04.243029 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:04.243003 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ffa3370-e4b4-45e6-86c2-23db0ea0b21d-kube-api-access-hn6wp" (OuterVolumeSpecName: "kube-api-access-hn6wp") pod "2ffa3370-e4b4-45e6-86c2-23db0ea0b21d" (UID: "2ffa3370-e4b4-45e6-86c2-23db0ea0b21d"). InnerVolumeSpecName "kube-api-access-hn6wp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 21:15:04.341766 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:04.341726 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn6wp\" (UniqueName: \"kubernetes.io/projected/2ffa3370-e4b4-45e6-86c2-23db0ea0b21d-kube-api-access-hn6wp\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:15:04.532995 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:04.532958 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-ftlx6"] Jul 20 21:15:04.537692 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:04.537667 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-ftlx6"] Jul 20 21:15:05.270104 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:05.270071 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ffa3370-e4b4-45e6-86c2-23db0ea0b21d" path="/var/lib/kubelet/pods/2ffa3370-e4b4-45e6-86c2-23db0ea0b21d/volumes" Jul 20 21:15:06.301929 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:06.301891 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:15:09.539701 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:09.539660 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:15:09.672827 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:09.672792 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:15:09.932585 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:09.932499 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:15:12.920849 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:12.920814 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:15:13.052676 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:13.052636 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:15:17.675253 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:17.675221 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:15:44.694841 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:44.694801 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:15:44.789653 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:44.789618 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:15:45.286581 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:45.286552 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:15:45.535894 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:45.535856 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:15:45.863835 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:45.863798 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:15:45.979243 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:45.979209 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:15:46.055354 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:46.055319 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-ktdwd"] Jul 20 21:15:46.055753 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:46.055739 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ffa3370-e4b4-45e6-86c2-23db0ea0b21d" containerName="authorino" Jul 20 21:15:46.055810 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:46.055754 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ffa3370-e4b4-45e6-86c2-23db0ea0b21d" containerName="authorino" Jul 20 21:15:46.055848 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:46.055814 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ffa3370-e4b4-45e6-86c2-23db0ea0b21d" containerName="authorino" Jul 20 21:15:46.058270 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:46.058251 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-ktdwd" Jul 20 21:15:46.062728 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:46.062703 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 20 21:15:46.062842 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:46.062791 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-5gl5w\"" Jul 20 21:15:46.078196 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:46.078170 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-ktdwd"] Jul 20 21:15:46.225965 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:46.225882 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sfmv\" (UniqueName: \"kubernetes.io/projected/937b1c81-87b3-4cba-8e7f-7cbf6aec77b6-kube-api-access-7sfmv\") pod \"payload-pre-processing-7677b774bb-ktdwd\" (UID: \"937b1c81-87b3-4cba-8e7f-7cbf6aec77b6\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-ktdwd" Jul 20 21:15:46.225965 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:46.225931 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/937b1c81-87b3-4cba-8e7f-7cbf6aec77b6-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-ktdwd\" (UID: \"937b1c81-87b3-4cba-8e7f-7cbf6aec77b6\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-ktdwd" Jul 20 21:15:46.327290 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:46.327258 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/937b1c81-87b3-4cba-8e7f-7cbf6aec77b6-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-ktdwd\" (UID: \"937b1c81-87b3-4cba-8e7f-7cbf6aec77b6\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-ktdwd" Jul 20 21:15:46.327459 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:46.327352 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sfmv\" (UniqueName: \"kubernetes.io/projected/937b1c81-87b3-4cba-8e7f-7cbf6aec77b6-kube-api-access-7sfmv\") pod \"payload-pre-processing-7677b774bb-ktdwd\" (UID: \"937b1c81-87b3-4cba-8e7f-7cbf6aec77b6\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-ktdwd" Jul 20 21:15:46.327960 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:46.327938 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/937b1c81-87b3-4cba-8e7f-7cbf6aec77b6-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-ktdwd\" (UID: \"937b1c81-87b3-4cba-8e7f-7cbf6aec77b6\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-ktdwd" Jul 20 21:15:46.338179 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:46.338123 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sfmv\" (UniqueName: \"kubernetes.io/projected/937b1c81-87b3-4cba-8e7f-7cbf6aec77b6-kube-api-access-7sfmv\") pod \"payload-pre-processing-7677b774bb-ktdwd\" (UID: \"937b1c81-87b3-4cba-8e7f-7cbf6aec77b6\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-ktdwd" Jul 20 21:15:46.368667 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:46.368633 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-ktdwd" Jul 20 21:15:46.530560 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:46.530536 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-ktdwd"] Jul 20 21:15:46.532263 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:15:46.532234 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod937b1c81_87b3_4cba_8e7f_7cbf6aec77b6.slice/crio-b9ad9b9a8ab8a80e5d76751627f32e1441be56ebe6c22352dba78d3751f41174 WatchSource:0}: Error finding container b9ad9b9a8ab8a80e5d76751627f32e1441be56ebe6c22352dba78d3751f41174: Status 404 returned error can't find the container with id b9ad9b9a8ab8a80e5d76751627f32e1441be56ebe6c22352dba78d3751f41174 Jul 20 21:15:47.423067 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:47.423021 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-ktdwd" event={"ID":"937b1c81-87b3-4cba-8e7f-7cbf6aec77b6","Type":"ContainerStarted","Data":"b9ad9b9a8ab8a80e5d76751627f32e1441be56ebe6c22352dba78d3751f41174"} Jul 20 21:15:48.429063 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:48.428973 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-ktdwd" event={"ID":"937b1c81-87b3-4cba-8e7f-7cbf6aec77b6","Type":"ContainerStarted","Data":"063837eee07906525ab38902ef4be8ee7bc8e8ac2935edb67f953f3c4aec59d1"} Jul 20 21:15:48.429063 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:48.429037 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-ktdwd" Jul 20 21:15:48.450169 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:48.450099 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7677b774bb-ktdwd" podStartSLOduration=0.836346327 podStartE2EDuration="2.450081462s" totalImagesPullingTime="1.613735135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 21:15:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:15:48.447113818 +0000 UTC m=+787.772816761" watchObservedRunningTime="2026-07-20 21:15:48.450081462 +0000 UTC m=+787.775784404" Jul 20 21:15:56.149587 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:56.149549 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:15:56.249374 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:56.249339 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:15:59.435815 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:15:59.435770 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-7677b774bb-ktdwd" podUID="937b1c81-87b3-4cba-8e7f-7cbf6aec77b6" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.33:9004: connect: connection refused" Jul 20 21:16:09.434219 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:09.434188 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-ktdwd" Jul 20 21:16:11.007262 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:11.007220 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:16:13.983017 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:13.982975 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-hflgl"] Jul 20 21:16:13.985848 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:13.985824 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hflgl" Jul 20 21:16:13.989268 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:13.989241 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 20 21:16:13.990311 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:13.990291 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 20 21:16:13.990449 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:13.990315 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-fmhkv\"" Jul 20 21:16:13.990449 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:13.990315 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 20 21:16:13.999256 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:13.999224 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-hflgl"] Jul 20 21:16:14.018935 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:14.018900 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:16:14.081653 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:14.081614 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqqb9\" (UniqueName: \"kubernetes.io/projected/158e2721-ac21-4cd4-b344-6adbf6afdc67-kube-api-access-lqqb9\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hflgl\" (UID: \"158e2721-ac21-4cd4-b344-6adbf6afdc67\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hflgl" Jul 20 21:16:14.081843 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:14.081674 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/158e2721-ac21-4cd4-b344-6adbf6afdc67-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hflgl\" (UID: \"158e2721-ac21-4cd4-b344-6adbf6afdc67\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hflgl" Jul 20 21:16:14.081843 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:14.081722 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/158e2721-ac21-4cd4-b344-6adbf6afdc67-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hflgl\" (UID: \"158e2721-ac21-4cd4-b344-6adbf6afdc67\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hflgl" Jul 20 21:16:14.081843 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:14.081762 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/158e2721-ac21-4cd4-b344-6adbf6afdc67-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hflgl\" (UID: \"158e2721-ac21-4cd4-b344-6adbf6afdc67\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hflgl" Jul 20 21:16:14.081843 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:14.081814 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/158e2721-ac21-4cd4-b344-6adbf6afdc67-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hflgl\" (UID: \"158e2721-ac21-4cd4-b344-6adbf6afdc67\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hflgl" Jul 20 21:16:14.081983 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:14.081877 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/158e2721-ac21-4cd4-b344-6adbf6afdc67-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hflgl\" (UID: \"158e2721-ac21-4cd4-b344-6adbf6afdc67\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hflgl" Jul 20 21:16:14.182736 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:14.182698 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/158e2721-ac21-4cd4-b344-6adbf6afdc67-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hflgl\" (UID: \"158e2721-ac21-4cd4-b344-6adbf6afdc67\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hflgl" Jul 20 21:16:14.182736 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:14.182745 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/158e2721-ac21-4cd4-b344-6adbf6afdc67-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hflgl\" (UID: \"158e2721-ac21-4cd4-b344-6adbf6afdc67\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hflgl" Jul 20 21:16:14.182959 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:14.182777 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/158e2721-ac21-4cd4-b344-6adbf6afdc67-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hflgl\" (UID: \"158e2721-ac21-4cd4-b344-6adbf6afdc67\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hflgl" Jul 20 21:16:14.183013 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:14.182950 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/158e2721-ac21-4cd4-b344-6adbf6afdc67-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hflgl\" (UID: \"158e2721-ac21-4cd4-b344-6adbf6afdc67\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hflgl" Jul 20 21:16:14.183054 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:14.183036 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/158e2721-ac21-4cd4-b344-6adbf6afdc67-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hflgl\" (UID: \"158e2721-ac21-4cd4-b344-6adbf6afdc67\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hflgl" Jul 20 21:16:14.183185 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:14.183165 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqqb9\" (UniqueName: \"kubernetes.io/projected/158e2721-ac21-4cd4-b344-6adbf6afdc67-kube-api-access-lqqb9\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hflgl\" (UID: \"158e2721-ac21-4cd4-b344-6adbf6afdc67\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hflgl" Jul 20 21:16:14.183267 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:14.183169 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/158e2721-ac21-4cd4-b344-6adbf6afdc67-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hflgl\" (UID: \"158e2721-ac21-4cd4-b344-6adbf6afdc67\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hflgl" Jul 20 21:16:14.183431 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:14.183384 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/158e2721-ac21-4cd4-b344-6adbf6afdc67-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hflgl\" (UID: \"158e2721-ac21-4cd4-b344-6adbf6afdc67\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hflgl" Jul 20 21:16:14.183431 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:14.183404 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/158e2721-ac21-4cd4-b344-6adbf6afdc67-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hflgl\" (UID: \"158e2721-ac21-4cd4-b344-6adbf6afdc67\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hflgl" Jul 20 21:16:14.185246 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:14.185230 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/158e2721-ac21-4cd4-b344-6adbf6afdc67-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hflgl\" (UID: \"158e2721-ac21-4cd4-b344-6adbf6afdc67\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hflgl" Jul 20 21:16:14.185517 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:14.185499 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/158e2721-ac21-4cd4-b344-6adbf6afdc67-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hflgl\" (UID: \"158e2721-ac21-4cd4-b344-6adbf6afdc67\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hflgl" Jul 20 21:16:14.191283 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:14.191262 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqqb9\" (UniqueName: \"kubernetes.io/projected/158e2721-ac21-4cd4-b344-6adbf6afdc67-kube-api-access-lqqb9\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hflgl\" (UID: \"158e2721-ac21-4cd4-b344-6adbf6afdc67\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hflgl" Jul 20 21:16:14.295827 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:14.295779 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hflgl" Jul 20 21:16:14.430014 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:14.429947 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-hflgl"] Jul 20 21:16:14.432465 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:16:14.432437 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod158e2721_ac21_4cd4_b344_6adbf6afdc67.slice/crio-b61198340b18e4c7feeaa1c6aeba09d2acacf47e558398fc68f189485886d5e5 WatchSource:0}: Error finding container b61198340b18e4c7feeaa1c6aeba09d2acacf47e558398fc68f189485886d5e5: Status 404 returned error can't find the container with id b61198340b18e4c7feeaa1c6aeba09d2acacf47e558398fc68f189485886d5e5 Jul 20 21:16:14.553102 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:14.553014 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hflgl" event={"ID":"158e2721-ac21-4cd4-b344-6adbf6afdc67","Type":"ContainerStarted","Data":"b61198340b18e4c7feeaa1c6aeba09d2acacf47e558398fc68f189485886d5e5"} Jul 20 21:16:15.560208 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:15.560171 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hflgl" event={"ID":"158e2721-ac21-4cd4-b344-6adbf6afdc67","Type":"ContainerStarted","Data":"d5e97cc861f2513d909dc947c3e07c2adb0e22a5035d90b2e6ef583938264b07"} Jul 20 21:16:15.582227 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:15.582170 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hflgl" podStartSLOduration=1.617692438 podStartE2EDuration="2.582127964s" totalImagesPullingTime="964.435526ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 21:16:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:16:15.580929016 +0000 UTC m=+814.906631959" watchObservedRunningTime="2026-07-20 21:16:15.582127964 +0000 UTC m=+814.907830907" Jul 20 21:16:21.888405 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:21.888363 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:16:24.296562 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:24.296519 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hflgl" Jul 20 21:16:24.297067 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:24.296594 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hflgl" Jul 20 21:16:24.309021 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:24.308991 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hflgl" Jul 20 21:16:24.614080 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:24.613984 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hflgl" Jul 20 21:16:29.081889 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:29.081852 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmwfzd"] Jul 20 21:16:29.085676 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:29.085652 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmwfzd" Jul 20 21:16:29.088438 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:29.088416 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 20 21:16:29.096506 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:29.096478 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmwfzd"] Jul 20 21:16:29.138444 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:29.138401 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:16:29.226922 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:29.226885 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xblc\" (UniqueName: \"kubernetes.io/projected/842a1a79-fcda-491d-a588-b364371d7648-kube-api-access-8xblc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmwfzd\" (UID: \"842a1a79-fcda-491d-a588-b364371d7648\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmwfzd" Jul 20 21:16:29.227102 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:29.226934 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/842a1a79-fcda-491d-a588-b364371d7648-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmwfzd\" (UID: \"842a1a79-fcda-491d-a588-b364371d7648\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmwfzd" Jul 20 21:16:29.227102 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:29.226961 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/842a1a79-fcda-491d-a588-b364371d7648-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmwfzd\" (UID: \"842a1a79-fcda-491d-a588-b364371d7648\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmwfzd" Jul 20 21:16:29.227102 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:29.226986 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/842a1a79-fcda-491d-a588-b364371d7648-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmwfzd\" (UID: \"842a1a79-fcda-491d-a588-b364371d7648\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmwfzd" Jul 20 21:16:29.227102 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:29.227010 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/842a1a79-fcda-491d-a588-b364371d7648-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmwfzd\" (UID: \"842a1a79-fcda-491d-a588-b364371d7648\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmwfzd" Jul 20 21:16:29.227102 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:29.227026 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/842a1a79-fcda-491d-a588-b364371d7648-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmwfzd\" (UID: \"842a1a79-fcda-491d-a588-b364371d7648\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmwfzd" Jul 20 21:16:29.328601 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:29.328563 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xblc\" (UniqueName: \"kubernetes.io/projected/842a1a79-fcda-491d-a588-b364371d7648-kube-api-access-8xblc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmwfzd\" (UID: \"842a1a79-fcda-491d-a588-b364371d7648\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmwfzd" Jul 20 21:16:29.328601 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:29.328614 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/842a1a79-fcda-491d-a588-b364371d7648-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmwfzd\" (UID: \"842a1a79-fcda-491d-a588-b364371d7648\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmwfzd" Jul 20 21:16:29.328864 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:29.328649 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/842a1a79-fcda-491d-a588-b364371d7648-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmwfzd\" (UID: \"842a1a79-fcda-491d-a588-b364371d7648\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmwfzd" Jul 20 21:16:29.328864 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:29.328683 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/842a1a79-fcda-491d-a588-b364371d7648-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmwfzd\" (UID: \"842a1a79-fcda-491d-a588-b364371d7648\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmwfzd" Jul 20 21:16:29.328864 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:29.328718 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/842a1a79-fcda-491d-a588-b364371d7648-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmwfzd\" (UID: \"842a1a79-fcda-491d-a588-b364371d7648\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmwfzd" Jul 20 21:16:29.329035 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:29.328861 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/842a1a79-fcda-491d-a588-b364371d7648-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmwfzd\" (UID: \"842a1a79-fcda-491d-a588-b364371d7648\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmwfzd" Jul 20 21:16:29.329110 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:29.329092 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/842a1a79-fcda-491d-a588-b364371d7648-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmwfzd\" (UID: \"842a1a79-fcda-491d-a588-b364371d7648\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmwfzd" Jul 20 21:16:29.329210 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:29.329184 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/842a1a79-fcda-491d-a588-b364371d7648-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmwfzd\" (UID: \"842a1a79-fcda-491d-a588-b364371d7648\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmwfzd" Jul 20 21:16:29.329300 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:29.329281 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/842a1a79-fcda-491d-a588-b364371d7648-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmwfzd\" (UID: \"842a1a79-fcda-491d-a588-b364371d7648\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmwfzd" Jul 20 21:16:29.331078 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:29.331051 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/842a1a79-fcda-491d-a588-b364371d7648-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmwfzd\" (UID: \"842a1a79-fcda-491d-a588-b364371d7648\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmwfzd" Jul 20 21:16:29.331365 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:29.331346 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/842a1a79-fcda-491d-a588-b364371d7648-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmwfzd\" (UID: \"842a1a79-fcda-491d-a588-b364371d7648\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmwfzd" Jul 20 21:16:29.336302 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:29.336246 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xblc\" (UniqueName: \"kubernetes.io/projected/842a1a79-fcda-491d-a588-b364371d7648-kube-api-access-8xblc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmwfzd\" (UID: \"842a1a79-fcda-491d-a588-b364371d7648\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmwfzd" Jul 20 21:16:29.398358 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:29.398317 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmwfzd" Jul 20 21:16:29.533602 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:29.533578 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmwfzd"] Jul 20 21:16:29.536167 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:16:29.536115 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod842a1a79_fcda_491d_a588_b364371d7648.slice/crio-d2cea57a190f7906b6532d945e2b3d8e19c3cb8853f45d314e795dd0b892e172 WatchSource:0}: Error finding container d2cea57a190f7906b6532d945e2b3d8e19c3cb8853f45d314e795dd0b892e172: Status 404 returned error can't find the container with id d2cea57a190f7906b6532d945e2b3d8e19c3cb8853f45d314e795dd0b892e172 Jul 20 21:16:29.627500 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:29.627397 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmwfzd" event={"ID":"842a1a79-fcda-491d-a588-b364371d7648","Type":"ContainerStarted","Data":"d2cea57a190f7906b6532d945e2b3d8e19c3cb8853f45d314e795dd0b892e172"} Jul 20 21:16:30.633339 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:30.633289 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmwfzd" event={"ID":"842a1a79-fcda-491d-a588-b364371d7648","Type":"ContainerStarted","Data":"6afd4f7be4ff855a702c82cb0b291cc9939439c65497873414a43ffd3a73d790"} Jul 20 21:16:30.654407 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:30.654347 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmwfzd" podStartSLOduration=1.483808351 podStartE2EDuration="1.654329795s" totalImagesPullingTime="170.521444ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 21:16:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:16:30.652306573 +0000 UTC m=+829.978009518" watchObservedRunningTime="2026-07-20 21:16:30.654329795 +0000 UTC m=+829.980032738" Jul 20 21:16:39.398918 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:39.398878 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmwfzd" Jul 20 21:16:39.399390 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:39.398955 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmwfzd" Jul 20 21:16:39.412233 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:39.412193 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmwfzd" Jul 20 21:16:39.687682 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:39.687585 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmwfzd" Jul 20 21:16:46.717928 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:46.717891 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:16:57.894126 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:57.894082 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-lw2sw"] Jul 20 21:16:57.898015 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:57.897987 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lw2sw" Jul 20 21:16:57.900700 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:57.900675 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 20 21:16:57.905591 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:57.905562 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-lw2sw"] Jul 20 21:16:57.980763 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:57.980732 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:16:57.986761 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:57.986739 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfe0744d-6ddd-473f-887b-6f1b5cc9f051-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lw2sw\" (UID: \"cfe0744d-6ddd-473f-887b-6f1b5cc9f051\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lw2sw" Jul 20 21:16:57.986861 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:57.986774 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfe0744d-6ddd-473f-887b-6f1b5cc9f051-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lw2sw\" (UID: \"cfe0744d-6ddd-473f-887b-6f1b5cc9f051\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lw2sw" Jul 20 21:16:57.986861 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:57.986796 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfe0744d-6ddd-473f-887b-6f1b5cc9f051-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lw2sw\" (UID: \"cfe0744d-6ddd-473f-887b-6f1b5cc9f051\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lw2sw" Jul 20 21:16:57.986861 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:57.986813 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfe0744d-6ddd-473f-887b-6f1b5cc9f051-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lw2sw\" (UID: \"cfe0744d-6ddd-473f-887b-6f1b5cc9f051\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lw2sw" Jul 20 21:16:57.987023 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:57.986905 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfe0744d-6ddd-473f-887b-6f1b5cc9f051-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lw2sw\" (UID: \"cfe0744d-6ddd-473f-887b-6f1b5cc9f051\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lw2sw" Jul 20 21:16:57.987023 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:57.986991 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pkjz\" (UniqueName: \"kubernetes.io/projected/cfe0744d-6ddd-473f-887b-6f1b5cc9f051-kube-api-access-2pkjz\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lw2sw\" (UID: \"cfe0744d-6ddd-473f-887b-6f1b5cc9f051\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lw2sw" Jul 20 21:16:58.087595 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:58.087557 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfe0744d-6ddd-473f-887b-6f1b5cc9f051-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lw2sw\" (UID: \"cfe0744d-6ddd-473f-887b-6f1b5cc9f051\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lw2sw" Jul 20 21:16:58.087595 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:58.087594 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfe0744d-6ddd-473f-887b-6f1b5cc9f051-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lw2sw\" (UID: \"cfe0744d-6ddd-473f-887b-6f1b5cc9f051\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lw2sw" Jul 20 21:16:58.087860 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:58.087613 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfe0744d-6ddd-473f-887b-6f1b5cc9f051-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lw2sw\" (UID: \"cfe0744d-6ddd-473f-887b-6f1b5cc9f051\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lw2sw" Jul 20 21:16:58.087860 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:58.087639 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfe0744d-6ddd-473f-887b-6f1b5cc9f051-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lw2sw\" (UID: \"cfe0744d-6ddd-473f-887b-6f1b5cc9f051\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lw2sw" Jul 20 21:16:58.087860 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:58.087675 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfe0744d-6ddd-473f-887b-6f1b5cc9f051-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lw2sw\" (UID: \"cfe0744d-6ddd-473f-887b-6f1b5cc9f051\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lw2sw" Jul 20 21:16:58.087860 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:58.087718 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pkjz\" (UniqueName: \"kubernetes.io/projected/cfe0744d-6ddd-473f-887b-6f1b5cc9f051-kube-api-access-2pkjz\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lw2sw\" (UID: \"cfe0744d-6ddd-473f-887b-6f1b5cc9f051\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lw2sw" Jul 20 21:16:58.088098 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:58.088071 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfe0744d-6ddd-473f-887b-6f1b5cc9f051-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lw2sw\" (UID: \"cfe0744d-6ddd-473f-887b-6f1b5cc9f051\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lw2sw" Jul 20 21:16:58.088195 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:58.088096 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfe0744d-6ddd-473f-887b-6f1b5cc9f051-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lw2sw\" (UID: \"cfe0744d-6ddd-473f-887b-6f1b5cc9f051\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lw2sw" Jul 20 21:16:58.088195 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:58.088129 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfe0744d-6ddd-473f-887b-6f1b5cc9f051-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lw2sw\" (UID: \"cfe0744d-6ddd-473f-887b-6f1b5cc9f051\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lw2sw" Jul 20 21:16:58.089810 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:58.089794 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfe0744d-6ddd-473f-887b-6f1b5cc9f051-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lw2sw\" (UID: \"cfe0744d-6ddd-473f-887b-6f1b5cc9f051\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lw2sw" Jul 20 21:16:58.090077 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:58.090057 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfe0744d-6ddd-473f-887b-6f1b5cc9f051-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lw2sw\" (UID: \"cfe0744d-6ddd-473f-887b-6f1b5cc9f051\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lw2sw" Jul 20 21:16:58.095308 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:58.095277 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pkjz\" (UniqueName: \"kubernetes.io/projected/cfe0744d-6ddd-473f-887b-6f1b5cc9f051-kube-api-access-2pkjz\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lw2sw\" (UID: \"cfe0744d-6ddd-473f-887b-6f1b5cc9f051\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lw2sw" Jul 20 21:16:58.208612 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:58.208508 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lw2sw" Jul 20 21:16:58.342483 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:58.342459 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-lw2sw"] Jul 20 21:16:58.344594 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:16:58.344566 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcfe0744d_6ddd_473f_887b_6f1b5cc9f051.slice/crio-e513b2032ac17894768a92c57137ce3c1b152bb165521ea77b99e8f6748e2d7f WatchSource:0}: Error finding container e513b2032ac17894768a92c57137ce3c1b152bb165521ea77b99e8f6748e2d7f: Status 404 returned error can't find the container with id e513b2032ac17894768a92c57137ce3c1b152bb165521ea77b99e8f6748e2d7f Jul 20 21:16:58.768597 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:58.768569 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lw2sw" event={"ID":"cfe0744d-6ddd-473f-887b-6f1b5cc9f051","Type":"ContainerStarted","Data":"e513b2032ac17894768a92c57137ce3c1b152bb165521ea77b99e8f6748e2d7f"} Jul 20 21:16:59.775452 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:59.775417 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lw2sw" event={"ID":"cfe0744d-6ddd-473f-887b-6f1b5cc9f051","Type":"ContainerStarted","Data":"5a5e1ac31744c92b433a8b7f04ab9e62fccb18fe5e1ae549d872a0be58becdc8"} Jul 20 21:16:59.792041 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:16:59.791986 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lw2sw" podStartSLOduration=2.421910485 podStartE2EDuration="2.791969888s" totalImagesPullingTime="370.059403ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 21:16:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:16:59.791404545 +0000 UTC m=+859.117107491" watchObservedRunningTime="2026-07-20 21:16:59.791969888 +0000 UTC m=+859.117672913" Jul 20 21:17:08.209241 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:17:08.209200 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lw2sw" Jul 20 21:17:08.209711 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:17:08.209281 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lw2sw" Jul 20 21:17:08.221667 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:17:08.221643 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lw2sw" Jul 20 21:17:08.826492 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:17:08.826463 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lw2sw" Jul 20 21:17:17.790986 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:17:17.790953 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:17:17.892686 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:17:17.892651 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:17:51.571277 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:17:51.571240 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:18:12.536055 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:18:12.536015 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:18:12.633015 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:18:12.632972 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:18:13.726061 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:18:13.726019 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:18:13.821021 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:18:13.820963 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:18:18.152466 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:18:18.152430 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:19:53.601211 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:19:53.601178 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:19:53.687335 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:19:53.687298 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:20:19.143299 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:20:19.143262 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:20:20.321592 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:20:20.321557 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:20:55.986158 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:20:55.986045 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:20:56.099426 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:20:56.099389 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:20:56.216262 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:20:56.216231 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:21:02.064415 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:21:02.064380 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:21:02.136292 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:21:02.136260 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:21:02.253557 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:21:02.253524 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:21:02.353438 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:21:02.353355 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:21:02.453054 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:21:02.453021 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:21:08.157525 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:21:08.157488 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:21:08.258606 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:21:08.258567 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:22:20.931999 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:22:20.931965 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:23:57.936472 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:23:57.936367 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:25:19.031952 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:25:19.031915 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:25:19.131080 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:25:19.131049 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:26:19.978982 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:26:19.978944 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:26:20.067230 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:26:20.067201 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:28:15.922755 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:28:15.922727 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:28:26.317843 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:28:26.317810 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:28:58.077061 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:28:58.077028 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc"] Jul 20 21:28:58.080454 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:28:58.080434 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc" Jul 20 21:28:58.085953 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:28:58.085922 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc\" is forbidden: User \"system:node:ip-10-0-136-32.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"llm\": no relationship found between node 'ip-10-0-136-32.ec2.internal' and this object" podUID="951c8882-b575-4c4a-bcca-81678a897b81" pod="llm/e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc" Jul 20 21:28:58.085953 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:28:58.085941 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"e2e-same-id-a-690af3af-kserve-self-signed-certs\" is forbidden: User \"system:node:ip-10-0-136-32.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"llm\": no relationship found between node 'ip-10-0-136-32.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"llm\"/\"e2e-same-id-a-690af3af-kserve-self-signed-certs\"" type="*v1.Secret" Jul 20 21:28:58.170755 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:28:58.170725 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc"] Jul 20 21:28:58.186020 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:28:58.185985 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:28:58.224553 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:28:58.224512 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/951c8882-b575-4c4a-bcca-81678a897b81-dshm\") pod \"e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc\" (UID: \"951c8882-b575-4c4a-bcca-81678a897b81\") " pod="llm/e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc" Jul 20 21:28:58.224701 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:28:58.224555 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/951c8882-b575-4c4a-bcca-81678a897b81-tmp-dir\") pod \"e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc\" (UID: \"951c8882-b575-4c4a-bcca-81678a897b81\") " pod="llm/e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc" Jul 20 21:28:58.224701 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:28:58.224583 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/951c8882-b575-4c4a-bcca-81678a897b81-tls-certs\") pod \"e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc\" (UID: \"951c8882-b575-4c4a-bcca-81678a897b81\") " pod="llm/e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc" Jul 20 21:28:58.224701 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:28:58.224655 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/951c8882-b575-4c4a-bcca-81678a897b81-model-cache\") pod \"e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc\" (UID: \"951c8882-b575-4c4a-bcca-81678a897b81\") " pod="llm/e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc" Jul 20 21:28:58.224701 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:28:58.224685 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/951c8882-b575-4c4a-bcca-81678a897b81-home\") pod \"e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc\" (UID: \"951c8882-b575-4c4a-bcca-81678a897b81\") " pod="llm/e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc" Jul 20 21:28:58.224838 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:28:58.224723 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlcdp\" (UniqueName: \"kubernetes.io/projected/951c8882-b575-4c4a-bcca-81678a897b81-kube-api-access-xlcdp\") pod \"e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc\" (UID: \"951c8882-b575-4c4a-bcca-81678a897b81\") " pod="llm/e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc" Jul 20 21:28:58.325316 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:28:58.325283 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlcdp\" (UniqueName: \"kubernetes.io/projected/951c8882-b575-4c4a-bcca-81678a897b81-kube-api-access-xlcdp\") pod \"e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc\" (UID: \"951c8882-b575-4c4a-bcca-81678a897b81\") " pod="llm/e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc" Jul 20 21:28:58.325316 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:28:58.325328 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/951c8882-b575-4c4a-bcca-81678a897b81-dshm\") pod \"e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc\" (UID: \"951c8882-b575-4c4a-bcca-81678a897b81\") " pod="llm/e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc" Jul 20 21:28:58.325532 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:28:58.325352 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/951c8882-b575-4c4a-bcca-81678a897b81-tmp-dir\") pod \"e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc\" (UID: \"951c8882-b575-4c4a-bcca-81678a897b81\") " pod="llm/e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc" Jul 20 21:28:58.325532 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:28:58.325373 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/951c8882-b575-4c4a-bcca-81678a897b81-tls-certs\") pod \"e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc\" (UID: \"951c8882-b575-4c4a-bcca-81678a897b81\") " pod="llm/e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc" Jul 20 21:28:58.325532 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:28:58.325436 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/951c8882-b575-4c4a-bcca-81678a897b81-model-cache\") pod \"e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc\" (UID: \"951c8882-b575-4c4a-bcca-81678a897b81\") " pod="llm/e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc" Jul 20 21:28:58.325532 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:28:58.325474 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/951c8882-b575-4c4a-bcca-81678a897b81-home\") pod \"e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc\" (UID: \"951c8882-b575-4c4a-bcca-81678a897b81\") " pod="llm/e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc" Jul 20 21:28:58.325779 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:28:58.325762 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/951c8882-b575-4c4a-bcca-81678a897b81-tmp-dir\") pod \"e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc\" (UID: \"951c8882-b575-4c4a-bcca-81678a897b81\") " pod="llm/e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc" Jul 20 21:28:58.325830 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:28:58.325782 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/951c8882-b575-4c4a-bcca-81678a897b81-home\") pod \"e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc\" (UID: \"951c8882-b575-4c4a-bcca-81678a897b81\") " pod="llm/e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc" Jul 20 21:28:58.325895 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:28:58.325876 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/951c8882-b575-4c4a-bcca-81678a897b81-model-cache\") pod \"e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc\" (UID: \"951c8882-b575-4c4a-bcca-81678a897b81\") " pod="llm/e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc" Jul 20 21:28:58.327506 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:28:58.327444 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/951c8882-b575-4c4a-bcca-81678a897b81-dshm\") pod \"e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc\" (UID: \"951c8882-b575-4c4a-bcca-81678a897b81\") " pod="llm/e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc" Jul 20 21:28:58.341283 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:28:58.341258 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlcdp\" (UniqueName: \"kubernetes.io/projected/951c8882-b575-4c4a-bcca-81678a897b81-kube-api-access-xlcdp\") pod \"e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc\" (UID: \"951c8882-b575-4c4a-bcca-81678a897b81\") " pod="llm/e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc" Jul 20 21:28:59.326296 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:28:59.326240 2562 secret.go:189] Couldn't get secret llm/e2e-same-id-a-690af3af-kserve-self-signed-certs: failed to sync secret cache: timed out waiting for the condition Jul 20 21:28:59.326811 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:28:59.326385 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/951c8882-b575-4c4a-bcca-81678a897b81-tls-certs podName:951c8882-b575-4c4a-bcca-81678a897b81 nodeName:}" failed. No retries permitted until 2026-07-20 21:28:59.82636291 +0000 UTC m=+1579.152065838 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/951c8882-b575-4c4a-bcca-81678a897b81-tls-certs") pod "e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc" (UID: "951c8882-b575-4c4a-bcca-81678a897b81") : failed to sync secret cache: timed out waiting for the condition Jul 20 21:28:59.684316 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:28:59.684239 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-690af3af-kserve-self-signed-certs\"" Jul 20 21:28:59.839036 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:28:59.839006 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/951c8882-b575-4c4a-bcca-81678a897b81-tls-certs\") pod \"e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc\" (UID: \"951c8882-b575-4c4a-bcca-81678a897b81\") " pod="llm/e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc" Jul 20 21:28:59.841570 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:28:59.841537 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/951c8882-b575-4c4a-bcca-81678a897b81-tls-certs\") pod \"e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc\" (UID: \"951c8882-b575-4c4a-bcca-81678a897b81\") " pod="llm/e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc" Jul 20 21:28:59.889432 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:28:59.889400 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc" Jul 20 21:29:00.045966 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:29:00.045944 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc"] Jul 20 21:29:00.047901 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:29:00.047872 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod951c8882_b575_4c4a_bcca_81678a897b81.slice/crio-2bd374138d98f9fa41243d6724f50e824516fff915c4d1a6ac86173674dfa054 WatchSource:0}: Error finding container 2bd374138d98f9fa41243d6724f50e824516fff915c4d1a6ac86173674dfa054: Status 404 returned error can't find the container with id 2bd374138d98f9fa41243d6724f50e824516fff915c4d1a6ac86173674dfa054 Jul 20 21:29:00.221193 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:29:00.221106 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc" event={"ID":"951c8882-b575-4c4a-bcca-81678a897b81","Type":"ContainerStarted","Data":"2acd7990fe545e900b695749ca4ce0eb9b358748674e47e16b7387714dde947d"} Jul 20 21:29:00.221193 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:29:00.221178 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc" event={"ID":"951c8882-b575-4c4a-bcca-81678a897b81","Type":"ContainerStarted","Data":"2bd374138d98f9fa41243d6724f50e824516fff915c4d1a6ac86173674dfa054"} Jul 20 21:29:06.348172 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:29:06.348102 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:29:09.890175 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:29:09.890112 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc" Jul 20 21:29:09.890175 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:29:09.890185 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc" Jul 20 21:29:09.902894 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:29:09.902857 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc" Jul 20 21:29:09.928065 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:29:09.928007 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc" podStartSLOduration=11.927989722 podStartE2EDuration="11.927989722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 21:28:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:29:00.262905873 +0000 UTC m=+1579.588608815" watchObservedRunningTime="2026-07-20 21:29:09.927989722 +0000 UTC m=+1589.253692664" Jul 20 21:29:10.284847 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:29:10.284818 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc" Jul 20 21:29:23.771726 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:29:23.771696 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:29:39.051023 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:29:39.050945 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc"] Jul 20 21:29:39.096926 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:29:39.096900 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:29:40.973897 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:29:40.973853 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:29:54.071225 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:29:54.071130 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc" podUID="951c8882-b575-4c4a-bcca-81678a897b81" containerName="main" containerID="cri-o://2acd7990fe545e900b695749ca4ce0eb9b358748674e47e16b7387714dde947d" gracePeriod=45 Jul 20 21:29:54.320483 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:29:54.320460 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc" Jul 20 21:29:54.431674 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:29:54.431560 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/951c8882-b575-4c4a-bcca-81678a897b81-tls-certs\") pod \"951c8882-b575-4c4a-bcca-81678a897b81\" (UID: \"951c8882-b575-4c4a-bcca-81678a897b81\") " Jul 20 21:29:54.431674 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:29:54.431655 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/951c8882-b575-4c4a-bcca-81678a897b81-dshm\") pod \"951c8882-b575-4c4a-bcca-81678a897b81\" (UID: \"951c8882-b575-4c4a-bcca-81678a897b81\") " Jul 20 21:29:54.431911 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:29:54.431752 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/951c8882-b575-4c4a-bcca-81678a897b81-tmp-dir\") pod \"951c8882-b575-4c4a-bcca-81678a897b81\" (UID: \"951c8882-b575-4c4a-bcca-81678a897b81\") " Jul 20 21:29:54.431911 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:29:54.431786 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/951c8882-b575-4c4a-bcca-81678a897b81-home\") pod \"951c8882-b575-4c4a-bcca-81678a897b81\" (UID: \"951c8882-b575-4c4a-bcca-81678a897b81\") " Jul 20 21:29:54.431911 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:29:54.431839 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/951c8882-b575-4c4a-bcca-81678a897b81-model-cache\") pod \"951c8882-b575-4c4a-bcca-81678a897b81\" (UID: \"951c8882-b575-4c4a-bcca-81678a897b81\") " Jul 20 21:29:54.431911 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:29:54.431856 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xlcdp\" (UniqueName: \"kubernetes.io/projected/951c8882-b575-4c4a-bcca-81678a897b81-kube-api-access-xlcdp\") pod \"951c8882-b575-4c4a-bcca-81678a897b81\" (UID: \"951c8882-b575-4c4a-bcca-81678a897b81\") " Jul 20 21:29:54.432080 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:29:54.432022 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/951c8882-b575-4c4a-bcca-81678a897b81-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "951c8882-b575-4c4a-bcca-81678a897b81" (UID: "951c8882-b575-4c4a-bcca-81678a897b81"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 21:29:54.432080 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:29:54.432066 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/951c8882-b575-4c4a-bcca-81678a897b81-home" (OuterVolumeSpecName: "home") pod "951c8882-b575-4c4a-bcca-81678a897b81" (UID: "951c8882-b575-4c4a-bcca-81678a897b81"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 21:29:54.432180 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:29:54.432078 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/951c8882-b575-4c4a-bcca-81678a897b81-model-cache" (OuterVolumeSpecName: "model-cache") pod "951c8882-b575-4c4a-bcca-81678a897b81" (UID: "951c8882-b575-4c4a-bcca-81678a897b81"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 21:29:54.432306 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:29:54.432285 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/951c8882-b575-4c4a-bcca-81678a897b81-tmp-dir\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:29:54.432367 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:29:54.432309 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/951c8882-b575-4c4a-bcca-81678a897b81-home\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:29:54.432367 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:29:54.432321 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/951c8882-b575-4c4a-bcca-81678a897b81-model-cache\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:29:54.433785 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:29:54.433762 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/951c8882-b575-4c4a-bcca-81678a897b81-dshm" (OuterVolumeSpecName: "dshm") pod "951c8882-b575-4c4a-bcca-81678a897b81" (UID: "951c8882-b575-4c4a-bcca-81678a897b81"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 21:29:54.433914 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:29:54.433890 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/951c8882-b575-4c4a-bcca-81678a897b81-kube-api-access-xlcdp" (OuterVolumeSpecName: "kube-api-access-xlcdp") pod "951c8882-b575-4c4a-bcca-81678a897b81" (UID: "951c8882-b575-4c4a-bcca-81678a897b81"). InnerVolumeSpecName "kube-api-access-xlcdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 21:29:54.434019 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:29:54.434005 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/951c8882-b575-4c4a-bcca-81678a897b81-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "951c8882-b575-4c4a-bcca-81678a897b81" (UID: "951c8882-b575-4c4a-bcca-81678a897b81"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 21:29:54.480039 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:29:54.480010 2562 generic.go:356] "Generic (PLEG): container finished" podID="951c8882-b575-4c4a-bcca-81678a897b81" containerID="2acd7990fe545e900b695749ca4ce0eb9b358748674e47e16b7387714dde947d" exitCode=0 Jul 20 21:29:54.480203 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:29:54.480055 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc" Jul 20 21:29:54.480203 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:29:54.480097 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc" event={"ID":"951c8882-b575-4c4a-bcca-81678a897b81","Type":"ContainerDied","Data":"2acd7990fe545e900b695749ca4ce0eb9b358748674e47e16b7387714dde947d"} Jul 20 21:29:54.480203 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:29:54.480152 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc" event={"ID":"951c8882-b575-4c4a-bcca-81678a897b81","Type":"ContainerDied","Data":"2bd374138d98f9fa41243d6724f50e824516fff915c4d1a6ac86173674dfa054"} Jul 20 21:29:54.480203 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:29:54.480172 2562 scope.go:117] "RemoveContainer" containerID="2acd7990fe545e900b695749ca4ce0eb9b358748674e47e16b7387714dde947d" Jul 20 21:29:54.489285 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:29:54.489267 2562 scope.go:117] "RemoveContainer" containerID="2acd7990fe545e900b695749ca4ce0eb9b358748674e47e16b7387714dde947d" Jul 20 21:29:54.489546 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:29:54.489525 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2acd7990fe545e900b695749ca4ce0eb9b358748674e47e16b7387714dde947d\": container with ID starting with 2acd7990fe545e900b695749ca4ce0eb9b358748674e47e16b7387714dde947d not found: ID does not exist" containerID="2acd7990fe545e900b695749ca4ce0eb9b358748674e47e16b7387714dde947d" Jul 20 21:29:54.489635 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:29:54.489557 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2acd7990fe545e900b695749ca4ce0eb9b358748674e47e16b7387714dde947d"} err="failed to get container status \"2acd7990fe545e900b695749ca4ce0eb9b358748674e47e16b7387714dde947d\": rpc error: code = NotFound desc = could not find container \"2acd7990fe545e900b695749ca4ce0eb9b358748674e47e16b7387714dde947d\": container with ID starting with 2acd7990fe545e900b695749ca4ce0eb9b358748674e47e16b7387714dde947d not found: ID does not exist" Jul 20 21:29:54.506340 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:29:54.506312 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc"] Jul 20 21:29:54.513763 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:29:54.513742 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-690af3af-kserve-5bdd7fd5b7-ntgtc"] Jul 20 21:29:54.533459 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:29:54.533438 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlcdp\" (UniqueName: \"kubernetes.io/projected/951c8882-b575-4c4a-bcca-81678a897b81-kube-api-access-xlcdp\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:29:54.533534 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:29:54.533462 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/951c8882-b575-4c4a-bcca-81678a897b81-tls-certs\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:29:54.533534 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:29:54.533473 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/951c8882-b575-4c4a-bcca-81678a897b81-dshm\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:29:55.271381 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:29:55.271349 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="951c8882-b575-4c4a-bcca-81678a897b81" path="/var/lib/kubelet/pods/951c8882-b575-4c4a-bcca-81678a897b81/volumes" Jul 20 21:32:50.763001 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:32:50.762956 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:33:00.394844 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:33:00.394805 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:33:00.501833 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:33:00.501797 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:33:02.556662 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:33:02.556623 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:33:10.811127 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:33:10.811087 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:33:16.044049 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:33:16.044007 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:33:19.256796 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:33:19.256764 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:33:19.440988 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:33:19.440946 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:33:19.731385 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:33:19.731340 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:33:25.758992 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:33:25.758943 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:33:25.857114 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:33:25.857078 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:33:25.998238 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:33:25.998209 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:33:26.329021 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:33:26.328982 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:33:26.583570 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:33:26.583465 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:33:29.249819 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:33:29.249768 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:33:29.343970 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:33:29.343937 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:33:29.438445 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:33:29.438397 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:33:29.746873 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:33:29.746834 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:33:29.935482 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:33:29.935447 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:33:30.169388 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:33:30.169348 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:34:05.602895 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:34:05.602797 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:34:10.092122 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:34:10.092077 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:34:11.359077 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:34:11.359044 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:34:11.489229 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:34:11.489191 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:34:11.591396 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:34:11.591359 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:34:13.695531 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:34:13.695494 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:34:14.045143 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:34:14.045114 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:34:14.396878 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:34:14.396786 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:34:14.800312 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:34:14.800278 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:34:14.900852 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:34:14.900812 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:34:49.357875 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:34:49.357843 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:34:53.708022 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:34:53.707982 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:34:53.938747 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:34:53.938709 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:34:54.107309 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:34:54.107264 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:34:54.366638 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:34:54.366549 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:34:54.692240 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:34:54.692127 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:34:54.897047 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:34:54.897016 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:34:56.802364 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:34:56.802326 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:34:56.899930 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:34:56.899893 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:34:56.982252 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:34:56.982220 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:34:57.215438 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:34:57.215344 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:34:57.646803 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:34:57.646755 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:34:57.769259 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:34:57.769227 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:35:33.124058 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:35:33.123975 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:35:37.749696 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:35:37.749665 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:35:38.970569 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:35:38.970535 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:35:39.191348 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:35:39.191318 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:35:39.447185 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:35:39.447154 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:35:39.794697 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:35:39.794660 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:35:40.043267 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:35:40.043227 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:35:41.986291 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:35:41.986249 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:35:42.084339 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:35:42.084290 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:35:42.197635 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:35:42.197606 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:35:42.597158 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:35:42.597104 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:35:42.778325 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:35:42.778286 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:35:42.876443 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:35:42.876350 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:35:44.297901 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:35:44.297869 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:35:44.406217 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:35:44.406179 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:36:03.745298 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:36:03.745255 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:36:04.221917 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:36:04.221878 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:36:17.338466 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:36:17.338429 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:36:21.933695 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:36:21.933658 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:36:22.098269 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:36:22.098229 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:36:22.345854 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:36:22.345822 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:36:22.818243 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:36:22.818200 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:36:23.003784 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:36:23.003754 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:36:23.128570 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:36:23.128478 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:36:24.531736 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:36:24.531691 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:36:30.300554 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:36:30.300499 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:36:30.402054 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:36:30.402008 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:36:30.525825 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:36:30.525797 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:36:30.615169 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:36:30.615064 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:36:31.053210 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:36:31.053176 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:36:57.098056 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:36:57.098020 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:37:04.667866 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:37:04.667780 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:37:04.825643 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:37:04.825612 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:37:04.950682 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:37:04.950598 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:37:20.927445 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:37:20.927413 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:37:27.913394 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:37:27.913358 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:37:28.094897 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:37:28.094864 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:37:38.674691 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:37:38.674649 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:37:39.541768 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:37:39.541737 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:37:46.522684 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:37:46.522649 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:37:46.631877 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:37:46.631841 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:37:46.726264 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:37:46.726221 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:37:46.890319 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:37:46.890232 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:37:53.066619 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:37:53.066574 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:37:53.155935 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:37:53.155900 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:37:53.266155 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:37:53.266091 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:37:55.071250 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:37:55.071210 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:38:01.955120 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:38:01.955082 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:38:02.075147 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:38:02.075104 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:38:02.836005 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:38:02.835969 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:38:10.491011 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:38:10.490974 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:38:10.625184 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:38:10.625129 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:38:10.791816 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:38:10.791782 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:38:11.285456 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:38:11.285417 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:38:11.760954 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:38:11.760908 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:38:14.508656 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:38:14.508623 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:38:14.780594 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:38:14.780496 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:38:15.038038 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:38:15.037939 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:38:16.947261 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:38:16.947225 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:38:17.054586 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:38:17.054551 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:38:18.830297 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:38:18.830246 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:38:19.576255 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:38:19.576215 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:38:19.707298 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:38:19.707262 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:38:20.134471 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:38:20.134433 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:38:26.206656 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:38:26.206625 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:38:26.286082 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:38:26.286041 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:38:37.249394 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:38:37.249306 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:38:41.681792 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:38:41.681754 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:38:42.092420 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:38:42.092389 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:38:48.295440 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:38:48.295393 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:38:48.398177 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:38:48.398143 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:38:48.582296 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:38:48.582214 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:38:48.843718 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:38:48.843639 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:38:49.025326 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:38:49.025297 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:38:49.118531 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:38:49.118447 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:38:49.232294 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:38:49.232254 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:38:55.019771 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:38:55.019729 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:38:55.126832 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:38:55.126795 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:38:55.233558 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:38:55.233524 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:38:55.447197 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:38:55.447085 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:39:01.620458 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:39:01.620411 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:39:01.697236 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:39:01.697202 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:39:08.589951 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:39:08.589912 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:39:08.948265 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:39:08.948179 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:39:09.169053 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:39:09.169022 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:39:11.395937 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:39:11.395908 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:39:13.257789 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:39:13.257753 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:39:15.905938 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:39:15.905903 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:39:28.287231 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:39:28.287200 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-bf0666db/e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t"] Jul 20 21:39:28.287616 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:39:28.287601 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="951c8882-b575-4c4a-bcca-81678a897b81" containerName="main" Jul 20 21:39:28.287616 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:39:28.287615 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="951c8882-b575-4c4a-bcca-81678a897b81" containerName="main" Jul 20 21:39:28.287695 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:39:28.287683 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="951c8882-b575-4c4a-bcca-81678a897b81" containerName="main" Jul 20 21:39:28.290585 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:39:28.290568 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-bf0666db/e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t" Jul 20 21:39:28.293209 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:39:28.293185 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-bf0666db\"/\"kube-root-ca.crt\"" Jul 20 21:39:28.294422 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:39:28.294402 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-bf0666db\"/\"e2e-lifecycle-model-bf0666db-kserve-self-signed-certs\"" Jul 20 21:39:28.294516 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:39:28.294464 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-bf0666db\"/\"openshift-service-ca.crt\"" Jul 20 21:39:28.294516 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:39:28.294472 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-bf0666db\"/\"default-dockercfg-kk2ll\"" Jul 20 21:39:28.301119 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:39:28.301094 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-bf0666db/e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t"] Jul 20 21:39:28.305975 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:39:28.305938 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7c22283-f46e-4157-996a-2e70e6aa8eb6-tls-certs\") pod \"e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t\" (UID: \"a7c22283-f46e-4157-996a-2e70e6aa8eb6\") " pod="ai-tenant-e2e-mt-bf0666db/e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t" Jul 20 21:39:28.306088 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:39:28.305979 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7c22283-f46e-4157-996a-2e70e6aa8eb6-model-cache\") pod \"e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t\" (UID: \"a7c22283-f46e-4157-996a-2e70e6aa8eb6\") " pod="ai-tenant-e2e-mt-bf0666db/e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t" Jul 20 21:39:28.306088 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:39:28.306010 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7c22283-f46e-4157-996a-2e70e6aa8eb6-home\") pod \"e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t\" (UID: \"a7c22283-f46e-4157-996a-2e70e6aa8eb6\") " pod="ai-tenant-e2e-mt-bf0666db/e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t" Jul 20 21:39:28.306240 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:39:28.306097 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46fc6\" (UniqueName: \"kubernetes.io/projected/a7c22283-f46e-4157-996a-2e70e6aa8eb6-kube-api-access-46fc6\") pod \"e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t\" (UID: \"a7c22283-f46e-4157-996a-2e70e6aa8eb6\") " pod="ai-tenant-e2e-mt-bf0666db/e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t" Jul 20 21:39:28.306240 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:39:28.306167 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7c22283-f46e-4157-996a-2e70e6aa8eb6-dshm\") pod \"e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t\" (UID: \"a7c22283-f46e-4157-996a-2e70e6aa8eb6\") " pod="ai-tenant-e2e-mt-bf0666db/e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t" Jul 20 21:39:28.306240 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:39:28.306199 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7c22283-f46e-4157-996a-2e70e6aa8eb6-tmp-dir\") pod \"e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t\" (UID: \"a7c22283-f46e-4157-996a-2e70e6aa8eb6\") " pod="ai-tenant-e2e-mt-bf0666db/e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t" Jul 20 21:39:28.393247 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:39:28.393220 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:39:28.407259 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:39:28.407227 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46fc6\" (UniqueName: \"kubernetes.io/projected/a7c22283-f46e-4157-996a-2e70e6aa8eb6-kube-api-access-46fc6\") pod \"e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t\" (UID: \"a7c22283-f46e-4157-996a-2e70e6aa8eb6\") " pod="ai-tenant-e2e-mt-bf0666db/e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t" Jul 20 21:39:28.407409 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:39:28.407275 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7c22283-f46e-4157-996a-2e70e6aa8eb6-dshm\") pod \"e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t\" (UID: \"a7c22283-f46e-4157-996a-2e70e6aa8eb6\") " pod="ai-tenant-e2e-mt-bf0666db/e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t" Jul 20 21:39:28.407409 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:39:28.407308 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7c22283-f46e-4157-996a-2e70e6aa8eb6-tmp-dir\") pod \"e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t\" (UID: \"a7c22283-f46e-4157-996a-2e70e6aa8eb6\") " pod="ai-tenant-e2e-mt-bf0666db/e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t" Jul 20 21:39:28.407409 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:39:28.407400 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7c22283-f46e-4157-996a-2e70e6aa8eb6-tls-certs\") pod \"e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t\" (UID: \"a7c22283-f46e-4157-996a-2e70e6aa8eb6\") " pod="ai-tenant-e2e-mt-bf0666db/e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t" Jul 20 21:39:28.407567 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:39:28.407439 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7c22283-f46e-4157-996a-2e70e6aa8eb6-model-cache\") pod \"e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t\" (UID: \"a7c22283-f46e-4157-996a-2e70e6aa8eb6\") " pod="ai-tenant-e2e-mt-bf0666db/e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t" Jul 20 21:39:28.407567 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:39:28.407469 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7c22283-f46e-4157-996a-2e70e6aa8eb6-home\") pod \"e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t\" (UID: \"a7c22283-f46e-4157-996a-2e70e6aa8eb6\") " pod="ai-tenant-e2e-mt-bf0666db/e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t" Jul 20 21:39:28.407925 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:39:28.407829 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7c22283-f46e-4157-996a-2e70e6aa8eb6-tmp-dir\") pod \"e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t\" (UID: \"a7c22283-f46e-4157-996a-2e70e6aa8eb6\") " pod="ai-tenant-e2e-mt-bf0666db/e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t" Jul 20 21:39:28.407925 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:39:28.407885 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7c22283-f46e-4157-996a-2e70e6aa8eb6-home\") pod \"e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t\" (UID: \"a7c22283-f46e-4157-996a-2e70e6aa8eb6\") " pod="ai-tenant-e2e-mt-bf0666db/e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t" Jul 20 21:39:28.407925 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:39:28.407903 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7c22283-f46e-4157-996a-2e70e6aa8eb6-model-cache\") pod \"e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t\" (UID: \"a7c22283-f46e-4157-996a-2e70e6aa8eb6\") " pod="ai-tenant-e2e-mt-bf0666db/e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t" Jul 20 21:39:28.409463 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:39:28.409443 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7c22283-f46e-4157-996a-2e70e6aa8eb6-dshm\") pod \"e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t\" (UID: \"a7c22283-f46e-4157-996a-2e70e6aa8eb6\") " pod="ai-tenant-e2e-mt-bf0666db/e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t" Jul 20 21:39:28.409744 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:39:28.409728 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7c22283-f46e-4157-996a-2e70e6aa8eb6-tls-certs\") pod \"e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t\" (UID: \"a7c22283-f46e-4157-996a-2e70e6aa8eb6\") " pod="ai-tenant-e2e-mt-bf0666db/e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t" Jul 20 21:39:28.415581 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:39:28.415556 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46fc6\" (UniqueName: \"kubernetes.io/projected/a7c22283-f46e-4157-996a-2e70e6aa8eb6-kube-api-access-46fc6\") pod \"e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t\" (UID: \"a7c22283-f46e-4157-996a-2e70e6aa8eb6\") " pod="ai-tenant-e2e-mt-bf0666db/e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t" Jul 20 21:39:28.600385 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:39:28.600291 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-bf0666db/e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t" Jul 20 21:39:28.737836 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:39:28.737811 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-bf0666db/e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t"] Jul 20 21:39:28.739599 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:39:28.739561 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7c22283_f46e_4157_996a_2e70e6aa8eb6.slice/crio-473a433f8fb111a89ef21632437a021a3b804d2421856f6fe97100ffa3c7650b WatchSource:0}: Error finding container 473a433f8fb111a89ef21632437a021a3b804d2421856f6fe97100ffa3c7650b: Status 404 returned error can't find the container with id 473a433f8fb111a89ef21632437a021a3b804d2421856f6fe97100ffa3c7650b Jul 20 21:39:29.225633 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:39:29.225602 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-bf0666db/e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t" event={"ID":"a7c22283-f46e-4157-996a-2e70e6aa8eb6","Type":"ContainerStarted","Data":"5265640ff02edb19cfbe952398cf095cb7520fb599d0f3063d2e2e818849ec1a"} Jul 20 21:39:29.225817 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:39:29.225645 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-bf0666db/e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t" event={"ID":"a7c22283-f46e-4157-996a-2e70e6aa8eb6","Type":"ContainerStarted","Data":"473a433f8fb111a89ef21632437a021a3b804d2421856f6fe97100ffa3c7650b"} Jul 20 21:39:29.245438 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:39:29.245382 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-bf0666db/e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t" podStartSLOduration=1.245366465 podStartE2EDuration="1.245366465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 21:39:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:39:29.243329288 +0000 UTC m=+2208.569032234" watchObservedRunningTime="2026-07-20 21:39:29.245366465 +0000 UTC m=+2208.571069408" Jul 20 21:39:38.600927 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:39:38.600889 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-bf0666db/e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t" Jul 20 21:39:38.601562 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:39:38.600950 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-bf0666db/e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t" Jul 20 21:39:38.614196 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:39:38.614171 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-bf0666db/e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t" Jul 20 21:39:39.285104 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:39:39.285071 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-bf0666db/e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t" Jul 20 21:40:05.151035 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:40:05.150953 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:40:09.625790 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:40:09.625759 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:40:11.943094 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:40:11.943063 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:40:16.798288 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:40:16.798257 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:40:16.919981 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:40:16.919945 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-bf0666db/e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t"] Jul 20 21:40:17.014249 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:40:17.014202 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:40:31.941031 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:40:31.940990 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-bf0666db/e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t" podUID="a7c22283-f46e-4157-996a-2e70e6aa8eb6" containerName="main" containerID="cri-o://5265640ff02edb19cfbe952398cf095cb7520fb599d0f3063d2e2e818849ec1a" gracePeriod=45 Jul 20 21:40:32.190589 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:40:32.190564 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-bf0666db/e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t" Jul 20 21:40:32.287393 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:40:32.287356 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7c22283-f46e-4157-996a-2e70e6aa8eb6-model-cache\") pod \"a7c22283-f46e-4157-996a-2e70e6aa8eb6\" (UID: \"a7c22283-f46e-4157-996a-2e70e6aa8eb6\") " Jul 20 21:40:32.287569 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:40:32.287413 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7c22283-f46e-4157-996a-2e70e6aa8eb6-home\") pod \"a7c22283-f46e-4157-996a-2e70e6aa8eb6\" (UID: \"a7c22283-f46e-4157-996a-2e70e6aa8eb6\") " Jul 20 21:40:32.287569 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:40:32.287434 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7c22283-f46e-4157-996a-2e70e6aa8eb6-tmp-dir\") pod \"a7c22283-f46e-4157-996a-2e70e6aa8eb6\" (UID: \"a7c22283-f46e-4157-996a-2e70e6aa8eb6\") " Jul 20 21:40:32.287569 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:40:32.287547 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46fc6\" (UniqueName: \"kubernetes.io/projected/a7c22283-f46e-4157-996a-2e70e6aa8eb6-kube-api-access-46fc6\") pod \"a7c22283-f46e-4157-996a-2e70e6aa8eb6\" (UID: \"a7c22283-f46e-4157-996a-2e70e6aa8eb6\") " Jul 20 21:40:32.287750 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:40:32.287581 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7c22283-f46e-4157-996a-2e70e6aa8eb6-tls-certs\") pod \"a7c22283-f46e-4157-996a-2e70e6aa8eb6\" (UID: \"a7c22283-f46e-4157-996a-2e70e6aa8eb6\") " Jul 20 21:40:32.287750 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:40:32.287611 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7c22283-f46e-4157-996a-2e70e6aa8eb6-dshm\") pod \"a7c22283-f46e-4157-996a-2e70e6aa8eb6\" (UID: \"a7c22283-f46e-4157-996a-2e70e6aa8eb6\") " Jul 20 21:40:32.287750 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:40:32.287656 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c22283-f46e-4157-996a-2e70e6aa8eb6-model-cache" (OuterVolumeSpecName: "model-cache") pod "a7c22283-f46e-4157-996a-2e70e6aa8eb6" (UID: "a7c22283-f46e-4157-996a-2e70e6aa8eb6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 21:40:32.287750 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:40:32.287701 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c22283-f46e-4157-996a-2e70e6aa8eb6-home" (OuterVolumeSpecName: "home") pod "a7c22283-f46e-4157-996a-2e70e6aa8eb6" (UID: "a7c22283-f46e-4157-996a-2e70e6aa8eb6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 21:40:32.287750 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:40:32.287734 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c22283-f46e-4157-996a-2e70e6aa8eb6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a7c22283-f46e-4157-996a-2e70e6aa8eb6" (UID: "a7c22283-f46e-4157-996a-2e70e6aa8eb6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 21:40:32.287994 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:40:32.287867 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7c22283-f46e-4157-996a-2e70e6aa8eb6-model-cache\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:40:32.287994 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:40:32.287879 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7c22283-f46e-4157-996a-2e70e6aa8eb6-home\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:40:32.287994 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:40:32.287887 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7c22283-f46e-4157-996a-2e70e6aa8eb6-tmp-dir\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:40:32.289720 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:40:32.289687 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c22283-f46e-4157-996a-2e70e6aa8eb6-dshm" (OuterVolumeSpecName: "dshm") pod "a7c22283-f46e-4157-996a-2e70e6aa8eb6" (UID: "a7c22283-f46e-4157-996a-2e70e6aa8eb6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 21:40:32.289837 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:40:32.289731 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7c22283-f46e-4157-996a-2e70e6aa8eb6-kube-api-access-46fc6" (OuterVolumeSpecName: "kube-api-access-46fc6") pod "a7c22283-f46e-4157-996a-2e70e6aa8eb6" (UID: "a7c22283-f46e-4157-996a-2e70e6aa8eb6"). InnerVolumeSpecName "kube-api-access-46fc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 21:40:32.289837 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:40:32.289763 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7c22283-f46e-4157-996a-2e70e6aa8eb6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a7c22283-f46e-4157-996a-2e70e6aa8eb6" (UID: "a7c22283-f46e-4157-996a-2e70e6aa8eb6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 21:40:32.389014 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:40:32.388984 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46fc6\" (UniqueName: \"kubernetes.io/projected/a7c22283-f46e-4157-996a-2e70e6aa8eb6-kube-api-access-46fc6\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:40:32.389014 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:40:32.389012 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7c22283-f46e-4157-996a-2e70e6aa8eb6-tls-certs\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:40:32.389236 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:40:32.389023 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7c22283-f46e-4157-996a-2e70e6aa8eb6-dshm\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:40:32.536704 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:40:32.536662 2562 generic.go:356] "Generic (PLEG): container finished" podID="a7c22283-f46e-4157-996a-2e70e6aa8eb6" containerID="5265640ff02edb19cfbe952398cf095cb7520fb599d0f3063d2e2e818849ec1a" exitCode=0 Jul 20 21:40:32.536908 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:40:32.536716 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-bf0666db/e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t" Jul 20 21:40:32.536908 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:40:32.536756 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-bf0666db/e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t" event={"ID":"a7c22283-f46e-4157-996a-2e70e6aa8eb6","Type":"ContainerDied","Data":"5265640ff02edb19cfbe952398cf095cb7520fb599d0f3063d2e2e818849ec1a"} Jul 20 21:40:32.536908 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:40:32.536804 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-bf0666db/e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t" event={"ID":"a7c22283-f46e-4157-996a-2e70e6aa8eb6","Type":"ContainerDied","Data":"473a433f8fb111a89ef21632437a021a3b804d2421856f6fe97100ffa3c7650b"} Jul 20 21:40:32.536908 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:40:32.536822 2562 scope.go:117] "RemoveContainer" containerID="5265640ff02edb19cfbe952398cf095cb7520fb599d0f3063d2e2e818849ec1a" Jul 20 21:40:32.545800 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:40:32.545782 2562 scope.go:117] "RemoveContainer" containerID="5265640ff02edb19cfbe952398cf095cb7520fb599d0f3063d2e2e818849ec1a" Jul 20 21:40:32.546038 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:40:32.546018 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5265640ff02edb19cfbe952398cf095cb7520fb599d0f3063d2e2e818849ec1a\": container with ID starting with 5265640ff02edb19cfbe952398cf095cb7520fb599d0f3063d2e2e818849ec1a not found: ID does not exist" containerID="5265640ff02edb19cfbe952398cf095cb7520fb599d0f3063d2e2e818849ec1a" Jul 20 21:40:32.546093 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:40:32.546052 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5265640ff02edb19cfbe952398cf095cb7520fb599d0f3063d2e2e818849ec1a"} err="failed to get container status \"5265640ff02edb19cfbe952398cf095cb7520fb599d0f3063d2e2e818849ec1a\": rpc error: code = NotFound desc = could not find container \"5265640ff02edb19cfbe952398cf095cb7520fb599d0f3063d2e2e818849ec1a\": container with ID starting with 5265640ff02edb19cfbe952398cf095cb7520fb599d0f3063d2e2e818849ec1a not found: ID does not exist" Jul 20 21:40:32.568103 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:40:32.568076 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-bf0666db/e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t"] Jul 20 21:40:32.573066 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:40:32.573040 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-bf0666db/e2e-lifecycle-model-bf0666db-kserve-759cb548f9-b565t"] Jul 20 21:40:33.272273 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:40:33.272238 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7c22283-f46e-4157-996a-2e70e6aa8eb6" path="/var/lib/kubelet/pods/a7c22283-f46e-4157-996a-2e70e6aa8eb6/volumes" Jul 20 21:40:45.224662 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:40:45.224625 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:40:45.328278 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:40:45.328242 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:40:53.573645 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:40:53.573610 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:40:54.615129 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:40:54.615093 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:41:00.320901 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:41:00.320859 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:41:00.421222 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:41:00.421180 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:41:00.605462 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:41:00.605364 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:41:00.852891 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:41:00.852857 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:41:01.961121 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:41:01.961095 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:41:02.082546 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:41:02.082510 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:41:02.183273 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:41:02.183234 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:41:02.935647 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:41:02.935588 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:41:03.029832 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:41:03.029798 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:41:03.111328 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:41:03.111297 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:41:04.569699 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:41:04.569655 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:41:10.426820 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:41:10.426785 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:41:10.527057 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:41:10.527017 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:41:11.120967 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:41:11.120937 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:41:11.290859 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:41:11.290821 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:41:17.331560 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:41:17.331526 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:41:17.444234 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:41:17.444197 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:41:17.562840 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:41:17.562806 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:41:28.081677 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:41:28.081643 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:41:28.416282 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:41:28.416189 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:41:38.328990 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:41:38.328886 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:41:38.429851 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:41:38.429818 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:41:49.414513 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:41:49.414471 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:41:56.769845 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:41:56.769800 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:41:56.860302 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:41:56.860262 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:41:57.080351 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:41:57.080272 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:41:57.406731 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:41:57.406646 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:41:57.823479 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:41:57.823443 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:42:00.215969 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:00.215937 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:42:00.308398 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:00.308362 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:42:00.385674 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:00.385631 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:42:00.572425 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:00.572394 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:42:01.065152 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:01.065097 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:42:01.220909 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:01.220877 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:42:04.753252 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:04.753206 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:42:05.184245 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:05.184214 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:42:05.587742 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:05.587709 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:42:05.811186 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:05.811127 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:42:07.342652 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:07.342618 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:42:08.767236 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:08.767197 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:42:08.941832 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:08.941793 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:42:09.484045 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:09.484004 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:42:09.583866 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:09.583828 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:42:09.683566 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:09.683532 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:42:13.222651 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:13.222612 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:42:22.196318 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:22.196277 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:42:22.296345 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:22.296318 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:42:31.337674 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:31.337642 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:42:31.438261 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:31.438226 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:42:33.590411 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:33.590378 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-b7f42a/test-model-b7f42a-kserve-8596847b49-hs87h"] Jul 20 21:42:33.590827 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:33.590776 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7c22283-f46e-4157-996a-2e70e6aa8eb6" containerName="main" Jul 20 21:42:33.590827 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:33.590788 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7c22283-f46e-4157-996a-2e70e6aa8eb6" containerName="main" Jul 20 21:42:33.590919 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:33.590851 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7c22283-f46e-4157-996a-2e70e6aa8eb6" containerName="main" Jul 20 21:42:33.593897 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:33.593877 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b7f42a/test-model-b7f42a-kserve-8596847b49-hs87h" Jul 20 21:42:33.597620 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:33.597597 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b7f42a\"/\"openshift-service-ca.crt\"" Jul 20 21:42:33.597729 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:33.597642 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b7f42a\"/\"kube-root-ca.crt\"" Jul 20 21:42:33.597729 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:33.597682 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b7f42a\"/\"default-dockercfg-lm5gj\"" Jul 20 21:42:33.597835 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:33.597646 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b7f42a\"/\"test-model-b7f42a-kserve-self-signed-certs\"" Jul 20 21:42:33.605869 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:33.605848 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b7f42a/test-model-b7f42a-kserve-8596847b49-hs87h"] Jul 20 21:42:33.627344 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:33.627315 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:42:33.721569 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:33.721535 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/706b6d30-8899-4a33-bfe0-f5d1324ec215-home\") pod \"test-model-b7f42a-kserve-8596847b49-hs87h\" (UID: \"706b6d30-8899-4a33-bfe0-f5d1324ec215\") " pod="ai-tenant-e2e-inf-a-b7f42a/test-model-b7f42a-kserve-8596847b49-hs87h" Jul 20 21:42:33.721754 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:33.721577 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/706b6d30-8899-4a33-bfe0-f5d1324ec215-dshm\") pod \"test-model-b7f42a-kserve-8596847b49-hs87h\" (UID: \"706b6d30-8899-4a33-bfe0-f5d1324ec215\") " pod="ai-tenant-e2e-inf-a-b7f42a/test-model-b7f42a-kserve-8596847b49-hs87h" Jul 20 21:42:33.721754 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:33.721595 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/706b6d30-8899-4a33-bfe0-f5d1324ec215-model-cache\") pod \"test-model-b7f42a-kserve-8596847b49-hs87h\" (UID: \"706b6d30-8899-4a33-bfe0-f5d1324ec215\") " pod="ai-tenant-e2e-inf-a-b7f42a/test-model-b7f42a-kserve-8596847b49-hs87h" Jul 20 21:42:33.721754 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:33.721661 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/706b6d30-8899-4a33-bfe0-f5d1324ec215-tmp-dir\") pod \"test-model-b7f42a-kserve-8596847b49-hs87h\" (UID: \"706b6d30-8899-4a33-bfe0-f5d1324ec215\") " pod="ai-tenant-e2e-inf-a-b7f42a/test-model-b7f42a-kserve-8596847b49-hs87h" Jul 20 21:42:33.721754 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:33.721697 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/706b6d30-8899-4a33-bfe0-f5d1324ec215-tls-certs\") pod \"test-model-b7f42a-kserve-8596847b49-hs87h\" (UID: \"706b6d30-8899-4a33-bfe0-f5d1324ec215\") " pod="ai-tenant-e2e-inf-a-b7f42a/test-model-b7f42a-kserve-8596847b49-hs87h" Jul 20 21:42:33.721754 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:33.721720 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g49fw\" (UniqueName: \"kubernetes.io/projected/706b6d30-8899-4a33-bfe0-f5d1324ec215-kube-api-access-g49fw\") pod \"test-model-b7f42a-kserve-8596847b49-hs87h\" (UID: \"706b6d30-8899-4a33-bfe0-f5d1324ec215\") " pod="ai-tenant-e2e-inf-a-b7f42a/test-model-b7f42a-kserve-8596847b49-hs87h" Jul 20 21:42:33.822894 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:33.822848 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/706b6d30-8899-4a33-bfe0-f5d1324ec215-home\") pod \"test-model-b7f42a-kserve-8596847b49-hs87h\" (UID: \"706b6d30-8899-4a33-bfe0-f5d1324ec215\") " pod="ai-tenant-e2e-inf-a-b7f42a/test-model-b7f42a-kserve-8596847b49-hs87h" Jul 20 21:42:33.823099 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:33.822906 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/706b6d30-8899-4a33-bfe0-f5d1324ec215-dshm\") pod \"test-model-b7f42a-kserve-8596847b49-hs87h\" (UID: \"706b6d30-8899-4a33-bfe0-f5d1324ec215\") " pod="ai-tenant-e2e-inf-a-b7f42a/test-model-b7f42a-kserve-8596847b49-hs87h" Jul 20 21:42:33.823099 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:33.822943 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/706b6d30-8899-4a33-bfe0-f5d1324ec215-model-cache\") pod \"test-model-b7f42a-kserve-8596847b49-hs87h\" (UID: \"706b6d30-8899-4a33-bfe0-f5d1324ec215\") " pod="ai-tenant-e2e-inf-a-b7f42a/test-model-b7f42a-kserve-8596847b49-hs87h" Jul 20 21:42:33.823099 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:33.822983 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/706b6d30-8899-4a33-bfe0-f5d1324ec215-tmp-dir\") pod \"test-model-b7f42a-kserve-8596847b49-hs87h\" (UID: \"706b6d30-8899-4a33-bfe0-f5d1324ec215\") " pod="ai-tenant-e2e-inf-a-b7f42a/test-model-b7f42a-kserve-8596847b49-hs87h" Jul 20 21:42:33.823099 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:33.823024 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/706b6d30-8899-4a33-bfe0-f5d1324ec215-tls-certs\") pod \"test-model-b7f42a-kserve-8596847b49-hs87h\" (UID: \"706b6d30-8899-4a33-bfe0-f5d1324ec215\") " pod="ai-tenant-e2e-inf-a-b7f42a/test-model-b7f42a-kserve-8596847b49-hs87h" Jul 20 21:42:33.823099 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:33.823057 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g49fw\" (UniqueName: \"kubernetes.io/projected/706b6d30-8899-4a33-bfe0-f5d1324ec215-kube-api-access-g49fw\") pod \"test-model-b7f42a-kserve-8596847b49-hs87h\" (UID: \"706b6d30-8899-4a33-bfe0-f5d1324ec215\") " pod="ai-tenant-e2e-inf-a-b7f42a/test-model-b7f42a-kserve-8596847b49-hs87h" Jul 20 21:42:33.823371 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:33.823347 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/706b6d30-8899-4a33-bfe0-f5d1324ec215-home\") pod \"test-model-b7f42a-kserve-8596847b49-hs87h\" (UID: \"706b6d30-8899-4a33-bfe0-f5d1324ec215\") " pod="ai-tenant-e2e-inf-a-b7f42a/test-model-b7f42a-kserve-8596847b49-hs87h" Jul 20 21:42:33.823424 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:33.823404 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/706b6d30-8899-4a33-bfe0-f5d1324ec215-tmp-dir\") pod \"test-model-b7f42a-kserve-8596847b49-hs87h\" (UID: \"706b6d30-8899-4a33-bfe0-f5d1324ec215\") " pod="ai-tenant-e2e-inf-a-b7f42a/test-model-b7f42a-kserve-8596847b49-hs87h" Jul 20 21:42:33.823461 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:33.823429 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/706b6d30-8899-4a33-bfe0-f5d1324ec215-model-cache\") pod \"test-model-b7f42a-kserve-8596847b49-hs87h\" (UID: \"706b6d30-8899-4a33-bfe0-f5d1324ec215\") " pod="ai-tenant-e2e-inf-a-b7f42a/test-model-b7f42a-kserve-8596847b49-hs87h" Jul 20 21:42:33.825320 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:33.825299 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/706b6d30-8899-4a33-bfe0-f5d1324ec215-dshm\") pod \"test-model-b7f42a-kserve-8596847b49-hs87h\" (UID: \"706b6d30-8899-4a33-bfe0-f5d1324ec215\") " pod="ai-tenant-e2e-inf-a-b7f42a/test-model-b7f42a-kserve-8596847b49-hs87h" Jul 20 21:42:33.825500 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:33.825484 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/706b6d30-8899-4a33-bfe0-f5d1324ec215-tls-certs\") pod \"test-model-b7f42a-kserve-8596847b49-hs87h\" (UID: \"706b6d30-8899-4a33-bfe0-f5d1324ec215\") " pod="ai-tenant-e2e-inf-a-b7f42a/test-model-b7f42a-kserve-8596847b49-hs87h" Jul 20 21:42:33.832514 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:33.832492 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g49fw\" (UniqueName: \"kubernetes.io/projected/706b6d30-8899-4a33-bfe0-f5d1324ec215-kube-api-access-g49fw\") pod \"test-model-b7f42a-kserve-8596847b49-hs87h\" (UID: \"706b6d30-8899-4a33-bfe0-f5d1324ec215\") " pod="ai-tenant-e2e-inf-a-b7f42a/test-model-b7f42a-kserve-8596847b49-hs87h" Jul 20 21:42:33.904589 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:33.904492 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b7f42a/test-model-b7f42a-kserve-8596847b49-hs87h" Jul 20 21:42:34.043086 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:34.043051 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b7f42a/test-model-b7f42a-kserve-8596847b49-hs87h"] Jul 20 21:42:34.045634 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:42:34.045603 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod706b6d30_8899_4a33_bfe0_f5d1324ec215.slice/crio-2ab948c07a5e79865df47b22a7ebca297034504cf1630dcdcda1c93f0f978311 WatchSource:0}: Error finding container 2ab948c07a5e79865df47b22a7ebca297034504cf1630dcdcda1c93f0f978311: Status 404 returned error can't find the container with id 2ab948c07a5e79865df47b22a7ebca297034504cf1630dcdcda1c93f0f978311 Jul 20 21:42:34.128757 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:34.128726 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b7f42a/test-model-b7f42a-kserve-8596847b49-hs87h" event={"ID":"706b6d30-8899-4a33-bfe0-f5d1324ec215","Type":"ContainerStarted","Data":"b84fb49d06e67326396fb4aa5c3aed4e56206ac31240394f6cf378c262af8ea9"} Jul 20 21:42:34.128925 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:34.128765 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b7f42a/test-model-b7f42a-kserve-8596847b49-hs87h" event={"ID":"706b6d30-8899-4a33-bfe0-f5d1324ec215","Type":"ContainerStarted","Data":"2ab948c07a5e79865df47b22a7ebca297034504cf1630dcdcda1c93f0f978311"} Jul 20 21:42:34.149095 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:34.149029 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-b7f42a/test-model-b7f42a-kserve-8596847b49-hs87h" podStartSLOduration=1.14899102 podStartE2EDuration="1.14899102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 21:42:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:42:34.148083814 +0000 UTC m=+2393.473786760" watchObservedRunningTime="2026-07-20 21:42:34.14899102 +0000 UTC m=+2393.474693968" Jul 20 21:42:43.905286 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:43.905248 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-b7f42a/test-model-b7f42a-kserve-8596847b49-hs87h" Jul 20 21:42:43.905688 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:43.905520 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-b7f42a/test-model-b7f42a-kserve-8596847b49-hs87h" Jul 20 21:42:43.917691 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:43.917667 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-b7f42a/test-model-b7f42a-kserve-8596847b49-hs87h" Jul 20 21:42:44.187630 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:44.187535 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-b7f42a/test-model-b7f42a-kserve-8596847b49-hs87h" Jul 20 21:42:57.389255 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:57.389219 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:42:57.490238 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:42:57.490198 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:43:09.008557 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:43:09.008534 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-2ad633/test-model-2ad633-kserve-6cbcdb9459-ltdm2"] Jul 20 21:43:09.012048 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:43:09.012029 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2ad633/test-model-2ad633-kserve-6cbcdb9459-ltdm2" Jul 20 21:43:09.015599 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:43:09.015575 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2ad633\"/\"default-dockercfg-qdfcd\"" Jul 20 21:43:09.015599 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:43:09.015596 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2ad633\"/\"kube-root-ca.crt\"" Jul 20 21:43:09.015795 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:43:09.015577 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2ad633\"/\"openshift-service-ca.crt\"" Jul 20 21:43:09.015795 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:43:09.015666 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2ad633\"/\"test-model-2ad633-kserve-self-signed-certs\"" Jul 20 21:43:09.024505 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:43:09.024483 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2ad633/test-model-2ad633-kserve-6cbcdb9459-ltdm2"] Jul 20 21:43:09.035689 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:43:09.035645 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f38e2042-5b60-4825-a7b0-1758413fa3ac-tmp-dir\") pod \"test-model-2ad633-kserve-6cbcdb9459-ltdm2\" (UID: \"f38e2042-5b60-4825-a7b0-1758413fa3ac\") " pod="ai-tenant-e2e-inf-b-2ad633/test-model-2ad633-kserve-6cbcdb9459-ltdm2" Jul 20 21:43:09.035790 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:43:09.035711 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f38e2042-5b60-4825-a7b0-1758413fa3ac-model-cache\") pod \"test-model-2ad633-kserve-6cbcdb9459-ltdm2\" (UID: \"f38e2042-5b60-4825-a7b0-1758413fa3ac\") " pod="ai-tenant-e2e-inf-b-2ad633/test-model-2ad633-kserve-6cbcdb9459-ltdm2" Jul 20 21:43:09.035790 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:43:09.035760 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f38e2042-5b60-4825-a7b0-1758413fa3ac-dshm\") pod \"test-model-2ad633-kserve-6cbcdb9459-ltdm2\" (UID: \"f38e2042-5b60-4825-a7b0-1758413fa3ac\") " pod="ai-tenant-e2e-inf-b-2ad633/test-model-2ad633-kserve-6cbcdb9459-ltdm2" Jul 20 21:43:09.035880 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:43:09.035858 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f38e2042-5b60-4825-a7b0-1758413fa3ac-tls-certs\") pod \"test-model-2ad633-kserve-6cbcdb9459-ltdm2\" (UID: \"f38e2042-5b60-4825-a7b0-1758413fa3ac\") " pod="ai-tenant-e2e-inf-b-2ad633/test-model-2ad633-kserve-6cbcdb9459-ltdm2" Jul 20 21:43:09.035926 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:43:09.035899 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fphqq\" (UniqueName: \"kubernetes.io/projected/f38e2042-5b60-4825-a7b0-1758413fa3ac-kube-api-access-fphqq\") pod \"test-model-2ad633-kserve-6cbcdb9459-ltdm2\" (UID: \"f38e2042-5b60-4825-a7b0-1758413fa3ac\") " pod="ai-tenant-e2e-inf-b-2ad633/test-model-2ad633-kserve-6cbcdb9459-ltdm2" Jul 20 21:43:09.035979 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:43:09.035923 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f38e2042-5b60-4825-a7b0-1758413fa3ac-home\") pod \"test-model-2ad633-kserve-6cbcdb9459-ltdm2\" (UID: \"f38e2042-5b60-4825-a7b0-1758413fa3ac\") " pod="ai-tenant-e2e-inf-b-2ad633/test-model-2ad633-kserve-6cbcdb9459-ltdm2" Jul 20 21:43:09.067516 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:43:09.067494 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:43:09.137421 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:43:09.137398 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f38e2042-5b60-4825-a7b0-1758413fa3ac-model-cache\") pod \"test-model-2ad633-kserve-6cbcdb9459-ltdm2\" (UID: \"f38e2042-5b60-4825-a7b0-1758413fa3ac\") " pod="ai-tenant-e2e-inf-b-2ad633/test-model-2ad633-kserve-6cbcdb9459-ltdm2" Jul 20 21:43:09.137538 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:43:09.137427 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f38e2042-5b60-4825-a7b0-1758413fa3ac-dshm\") pod \"test-model-2ad633-kserve-6cbcdb9459-ltdm2\" (UID: \"f38e2042-5b60-4825-a7b0-1758413fa3ac\") " pod="ai-tenant-e2e-inf-b-2ad633/test-model-2ad633-kserve-6cbcdb9459-ltdm2" Jul 20 21:43:09.137538 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:43:09.137451 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f38e2042-5b60-4825-a7b0-1758413fa3ac-tls-certs\") pod \"test-model-2ad633-kserve-6cbcdb9459-ltdm2\" (UID: \"f38e2042-5b60-4825-a7b0-1758413fa3ac\") " pod="ai-tenant-e2e-inf-b-2ad633/test-model-2ad633-kserve-6cbcdb9459-ltdm2" Jul 20 21:43:09.137615 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:43:09.137574 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fphqq\" (UniqueName: \"kubernetes.io/projected/f38e2042-5b60-4825-a7b0-1758413fa3ac-kube-api-access-fphqq\") pod \"test-model-2ad633-kserve-6cbcdb9459-ltdm2\" (UID: \"f38e2042-5b60-4825-a7b0-1758413fa3ac\") " pod="ai-tenant-e2e-inf-b-2ad633/test-model-2ad633-kserve-6cbcdb9459-ltdm2" Jul 20 21:43:09.137665 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:43:09.137619 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f38e2042-5b60-4825-a7b0-1758413fa3ac-home\") pod \"test-model-2ad633-kserve-6cbcdb9459-ltdm2\" (UID: \"f38e2042-5b60-4825-a7b0-1758413fa3ac\") " pod="ai-tenant-e2e-inf-b-2ad633/test-model-2ad633-kserve-6cbcdb9459-ltdm2" Jul 20 21:43:09.137766 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:43:09.137733 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f38e2042-5b60-4825-a7b0-1758413fa3ac-tmp-dir\") pod \"test-model-2ad633-kserve-6cbcdb9459-ltdm2\" (UID: \"f38e2042-5b60-4825-a7b0-1758413fa3ac\") " pod="ai-tenant-e2e-inf-b-2ad633/test-model-2ad633-kserve-6cbcdb9459-ltdm2" Jul 20 21:43:09.137841 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:43:09.137783 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f38e2042-5b60-4825-a7b0-1758413fa3ac-model-cache\") pod \"test-model-2ad633-kserve-6cbcdb9459-ltdm2\" (UID: \"f38e2042-5b60-4825-a7b0-1758413fa3ac\") " pod="ai-tenant-e2e-inf-b-2ad633/test-model-2ad633-kserve-6cbcdb9459-ltdm2" Jul 20 21:43:09.138009 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:43:09.137990 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f38e2042-5b60-4825-a7b0-1758413fa3ac-home\") pod \"test-model-2ad633-kserve-6cbcdb9459-ltdm2\" (UID: \"f38e2042-5b60-4825-a7b0-1758413fa3ac\") " pod="ai-tenant-e2e-inf-b-2ad633/test-model-2ad633-kserve-6cbcdb9459-ltdm2" Jul 20 21:43:09.138102 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:43:09.138012 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f38e2042-5b60-4825-a7b0-1758413fa3ac-tmp-dir\") pod \"test-model-2ad633-kserve-6cbcdb9459-ltdm2\" (UID: \"f38e2042-5b60-4825-a7b0-1758413fa3ac\") " pod="ai-tenant-e2e-inf-b-2ad633/test-model-2ad633-kserve-6cbcdb9459-ltdm2" Jul 20 21:43:09.139704 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:43:09.139676 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f38e2042-5b60-4825-a7b0-1758413fa3ac-dshm\") pod \"test-model-2ad633-kserve-6cbcdb9459-ltdm2\" (UID: \"f38e2042-5b60-4825-a7b0-1758413fa3ac\") " pod="ai-tenant-e2e-inf-b-2ad633/test-model-2ad633-kserve-6cbcdb9459-ltdm2" Jul 20 21:43:09.139883 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:43:09.139865 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f38e2042-5b60-4825-a7b0-1758413fa3ac-tls-certs\") pod \"test-model-2ad633-kserve-6cbcdb9459-ltdm2\" (UID: \"f38e2042-5b60-4825-a7b0-1758413fa3ac\") " pod="ai-tenant-e2e-inf-b-2ad633/test-model-2ad633-kserve-6cbcdb9459-ltdm2" Jul 20 21:43:09.146426 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:43:09.146406 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fphqq\" (UniqueName: \"kubernetes.io/projected/f38e2042-5b60-4825-a7b0-1758413fa3ac-kube-api-access-fphqq\") pod \"test-model-2ad633-kserve-6cbcdb9459-ltdm2\" (UID: \"f38e2042-5b60-4825-a7b0-1758413fa3ac\") " pod="ai-tenant-e2e-inf-b-2ad633/test-model-2ad633-kserve-6cbcdb9459-ltdm2" Jul 20 21:43:09.321520 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:43:09.321440 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2ad633/test-model-2ad633-kserve-6cbcdb9459-ltdm2" Jul 20 21:43:09.461180 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:43:09.461155 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2ad633/test-model-2ad633-kserve-6cbcdb9459-ltdm2"] Jul 20 21:43:09.463021 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:43:09.462994 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf38e2042_5b60_4825_a7b0_1758413fa3ac.slice/crio-014b902fd9b6d7ad851bd6462757159352e6cc0d8ed46599bb9b6e5907fff9c1 WatchSource:0}: Error finding container 014b902fd9b6d7ad851bd6462757159352e6cc0d8ed46599bb9b6e5907fff9c1: Status 404 returned error can't find the container with id 014b902fd9b6d7ad851bd6462757159352e6cc0d8ed46599bb9b6e5907fff9c1 Jul 20 21:43:10.305149 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:43:10.305106 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2ad633/test-model-2ad633-kserve-6cbcdb9459-ltdm2" event={"ID":"f38e2042-5b60-4825-a7b0-1758413fa3ac","Type":"ContainerStarted","Data":"04bcf4b9a691691342e0722469248238238ac34755708388aa9523888c406d9a"} Jul 20 21:43:10.305535 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:43:10.305164 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2ad633/test-model-2ad633-kserve-6cbcdb9459-ltdm2" event={"ID":"f38e2042-5b60-4825-a7b0-1758413fa3ac","Type":"ContainerStarted","Data":"014b902fd9b6d7ad851bd6462757159352e6cc0d8ed46599bb9b6e5907fff9c1"} Jul 20 21:43:10.325176 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:43:10.325111 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-2ad633/test-model-2ad633-kserve-6cbcdb9459-ltdm2" podStartSLOduration=2.325093714 podStartE2EDuration="2.325093714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 21:43:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:43:10.323068122 +0000 UTC m=+2429.648771075" watchObservedRunningTime="2026-07-20 21:43:10.325093714 +0000 UTC m=+2429.650796655" Jul 20 21:43:19.322212 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:43:19.322173 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-2ad633/test-model-2ad633-kserve-6cbcdb9459-ltdm2" Jul 20 21:43:19.322664 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:43:19.322231 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-2ad633/test-model-2ad633-kserve-6cbcdb9459-ltdm2" Jul 20 21:43:19.334656 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:43:19.334623 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-2ad633/test-model-2ad633-kserve-6cbcdb9459-ltdm2" Jul 20 21:43:19.364167 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:43:19.364114 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-2ad633/test-model-2ad633-kserve-6cbcdb9459-ltdm2" Jul 20 21:43:53.642770 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:43:53.642735 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b7f42a/test-model-b7f42a-kserve-8596847b49-hs87h"] Jul 20 21:43:53.679247 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:43:53.679219 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:43:54.529406 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:43:54.529374 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2ad633/test-model-2ad633-kserve-6cbcdb9459-ltdm2"] Jul 20 21:43:54.553837 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:43:54.553800 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:43:55.523356 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:43:55.523312 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:43:55.625173 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:43:55.625115 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:44:05.152549 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:05.152516 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:44:08.662590 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:08.662473 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-b7f42a/test-model-b7f42a-kserve-8596847b49-hs87h" podUID="706b6d30-8899-4a33-bfe0-f5d1324ec215" containerName="main" containerID="cri-o://b84fb49d06e67326396fb4aa5c3aed4e56206ac31240394f6cf378c262af8ea9" gracePeriod=45 Jul 20 21:44:08.918624 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:08.918565 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b7f42a/test-model-b7f42a-kserve-8596847b49-hs87h" Jul 20 21:44:09.010003 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:09.009968 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g49fw\" (UniqueName: \"kubernetes.io/projected/706b6d30-8899-4a33-bfe0-f5d1324ec215-kube-api-access-g49fw\") pod \"706b6d30-8899-4a33-bfe0-f5d1324ec215\" (UID: \"706b6d30-8899-4a33-bfe0-f5d1324ec215\") " Jul 20 21:44:09.010204 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:09.010015 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/706b6d30-8899-4a33-bfe0-f5d1324ec215-dshm\") pod \"706b6d30-8899-4a33-bfe0-f5d1324ec215\" (UID: \"706b6d30-8899-4a33-bfe0-f5d1324ec215\") " Jul 20 21:44:09.010204 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:09.010063 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/706b6d30-8899-4a33-bfe0-f5d1324ec215-model-cache\") pod \"706b6d30-8899-4a33-bfe0-f5d1324ec215\" (UID: \"706b6d30-8899-4a33-bfe0-f5d1324ec215\") " Jul 20 21:44:09.010204 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:09.010107 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/706b6d30-8899-4a33-bfe0-f5d1324ec215-home\") pod \"706b6d30-8899-4a33-bfe0-f5d1324ec215\" (UID: \"706b6d30-8899-4a33-bfe0-f5d1324ec215\") " Jul 20 21:44:09.010204 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:09.010182 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/706b6d30-8899-4a33-bfe0-f5d1324ec215-tls-certs\") pod \"706b6d30-8899-4a33-bfe0-f5d1324ec215\" (UID: \"706b6d30-8899-4a33-bfe0-f5d1324ec215\") " Jul 20 21:44:09.010439 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:09.010247 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/706b6d30-8899-4a33-bfe0-f5d1324ec215-tmp-dir\") pod \"706b6d30-8899-4a33-bfe0-f5d1324ec215\" (UID: \"706b6d30-8899-4a33-bfe0-f5d1324ec215\") " Jul 20 21:44:09.010439 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:09.010365 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/706b6d30-8899-4a33-bfe0-f5d1324ec215-model-cache" (OuterVolumeSpecName: "model-cache") pod "706b6d30-8899-4a33-bfe0-f5d1324ec215" (UID: "706b6d30-8899-4a33-bfe0-f5d1324ec215"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 21:44:09.010439 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:09.010380 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/706b6d30-8899-4a33-bfe0-f5d1324ec215-home" (OuterVolumeSpecName: "home") pod "706b6d30-8899-4a33-bfe0-f5d1324ec215" (UID: "706b6d30-8899-4a33-bfe0-f5d1324ec215"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 21:44:09.010565 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:09.010528 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/706b6d30-8899-4a33-bfe0-f5d1324ec215-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "706b6d30-8899-4a33-bfe0-f5d1324ec215" (UID: "706b6d30-8899-4a33-bfe0-f5d1324ec215"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 21:44:09.010623 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:09.010563 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/706b6d30-8899-4a33-bfe0-f5d1324ec215-home\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:44:09.010623 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:09.010577 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/706b6d30-8899-4a33-bfe0-f5d1324ec215-model-cache\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:44:09.012308 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:09.012280 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/706b6d30-8899-4a33-bfe0-f5d1324ec215-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "706b6d30-8899-4a33-bfe0-f5d1324ec215" (UID: "706b6d30-8899-4a33-bfe0-f5d1324ec215"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 21:44:09.012432 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:09.012308 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/706b6d30-8899-4a33-bfe0-f5d1324ec215-dshm" (OuterVolumeSpecName: "dshm") pod "706b6d30-8899-4a33-bfe0-f5d1324ec215" (UID: "706b6d30-8899-4a33-bfe0-f5d1324ec215"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 21:44:09.012432 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:09.012306 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/706b6d30-8899-4a33-bfe0-f5d1324ec215-kube-api-access-g49fw" (OuterVolumeSpecName: "kube-api-access-g49fw") pod "706b6d30-8899-4a33-bfe0-f5d1324ec215" (UID: "706b6d30-8899-4a33-bfe0-f5d1324ec215"). InnerVolumeSpecName "kube-api-access-g49fw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 21:44:09.111579 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:09.111544 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/706b6d30-8899-4a33-bfe0-f5d1324ec215-tmp-dir\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:44:09.111579 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:09.111577 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g49fw\" (UniqueName: \"kubernetes.io/projected/706b6d30-8899-4a33-bfe0-f5d1324ec215-kube-api-access-g49fw\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:44:09.111773 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:09.111591 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/706b6d30-8899-4a33-bfe0-f5d1324ec215-dshm\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:44:09.111773 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:09.111603 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/706b6d30-8899-4a33-bfe0-f5d1324ec215-tls-certs\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:44:09.549457 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:09.549369 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-2ad633/test-model-2ad633-kserve-6cbcdb9459-ltdm2" podUID="f38e2042-5b60-4825-a7b0-1758413fa3ac" containerName="main" containerID="cri-o://04bcf4b9a691691342e0722469248238238ac34755708388aa9523888c406d9a" gracePeriod=45 Jul 20 21:44:09.588418 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:09.588385 2562 generic.go:356] "Generic (PLEG): container finished" podID="706b6d30-8899-4a33-bfe0-f5d1324ec215" containerID="b84fb49d06e67326396fb4aa5c3aed4e56206ac31240394f6cf378c262af8ea9" exitCode=0 Jul 20 21:44:09.588560 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:09.588434 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b7f42a/test-model-b7f42a-kserve-8596847b49-hs87h" Jul 20 21:44:09.588560 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:09.588476 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b7f42a/test-model-b7f42a-kserve-8596847b49-hs87h" event={"ID":"706b6d30-8899-4a33-bfe0-f5d1324ec215","Type":"ContainerDied","Data":"b84fb49d06e67326396fb4aa5c3aed4e56206ac31240394f6cf378c262af8ea9"} Jul 20 21:44:09.588560 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:09.588518 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b7f42a/test-model-b7f42a-kserve-8596847b49-hs87h" event={"ID":"706b6d30-8899-4a33-bfe0-f5d1324ec215","Type":"ContainerDied","Data":"2ab948c07a5e79865df47b22a7ebca297034504cf1630dcdcda1c93f0f978311"} Jul 20 21:44:09.588560 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:09.588535 2562 scope.go:117] "RemoveContainer" containerID="b84fb49d06e67326396fb4aa5c3aed4e56206ac31240394f6cf378c262af8ea9" Jul 20 21:44:09.628293 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:09.628267 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b7f42a/test-model-b7f42a-kserve-8596847b49-hs87h"] Jul 20 21:44:09.632792 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:09.632767 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-b7f42a/test-model-b7f42a-kserve-8596847b49-hs87h"] Jul 20 21:44:09.659662 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:09.659644 2562 scope.go:117] "RemoveContainer" containerID="b84fb49d06e67326396fb4aa5c3aed4e56206ac31240394f6cf378c262af8ea9" Jul 20 21:44:09.659995 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:44:09.659977 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b84fb49d06e67326396fb4aa5c3aed4e56206ac31240394f6cf378c262af8ea9\": container with ID starting with b84fb49d06e67326396fb4aa5c3aed4e56206ac31240394f6cf378c262af8ea9 not found: ID does not exist" containerID="b84fb49d06e67326396fb4aa5c3aed4e56206ac31240394f6cf378c262af8ea9" Jul 20 21:44:09.660052 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:09.660003 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b84fb49d06e67326396fb4aa5c3aed4e56206ac31240394f6cf378c262af8ea9"} err="failed to get container status \"b84fb49d06e67326396fb4aa5c3aed4e56206ac31240394f6cf378c262af8ea9\": rpc error: code = NotFound desc = could not find container \"b84fb49d06e67326396fb4aa5c3aed4e56206ac31240394f6cf378c262af8ea9\": container with ID starting with b84fb49d06e67326396fb4aa5c3aed4e56206ac31240394f6cf378c262af8ea9 not found: ID does not exist" Jul 20 21:44:09.751670 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:09.751610 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:44:09.806474 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:09.806407 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2ad633/test-model-2ad633-kserve-6cbcdb9459-ltdm2" Jul 20 21:44:09.919233 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:09.919199 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f38e2042-5b60-4825-a7b0-1758413fa3ac-model-cache\") pod \"f38e2042-5b60-4825-a7b0-1758413fa3ac\" (UID: \"f38e2042-5b60-4825-a7b0-1758413fa3ac\") " Jul 20 21:44:09.919414 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:09.919247 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f38e2042-5b60-4825-a7b0-1758413fa3ac-dshm\") pod \"f38e2042-5b60-4825-a7b0-1758413fa3ac\" (UID: \"f38e2042-5b60-4825-a7b0-1758413fa3ac\") " Jul 20 21:44:09.919414 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:09.919321 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f38e2042-5b60-4825-a7b0-1758413fa3ac-tls-certs\") pod \"f38e2042-5b60-4825-a7b0-1758413fa3ac\" (UID: \"f38e2042-5b60-4825-a7b0-1758413fa3ac\") " Jul 20 21:44:09.919414 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:09.919349 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f38e2042-5b60-4825-a7b0-1758413fa3ac-tmp-dir\") pod \"f38e2042-5b60-4825-a7b0-1758413fa3ac\" (UID: \"f38e2042-5b60-4825-a7b0-1758413fa3ac\") " Jul 20 21:44:09.919414 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:09.919403 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f38e2042-5b60-4825-a7b0-1758413fa3ac-home\") pod \"f38e2042-5b60-4825-a7b0-1758413fa3ac\" (UID: \"f38e2042-5b60-4825-a7b0-1758413fa3ac\") " Jul 20 21:44:09.919650 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:09.919467 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fphqq\" (UniqueName: \"kubernetes.io/projected/f38e2042-5b60-4825-a7b0-1758413fa3ac-kube-api-access-fphqq\") pod \"f38e2042-5b60-4825-a7b0-1758413fa3ac\" (UID: \"f38e2042-5b60-4825-a7b0-1758413fa3ac\") " Jul 20 21:44:09.919650 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:09.919565 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f38e2042-5b60-4825-a7b0-1758413fa3ac-model-cache" (OuterVolumeSpecName: "model-cache") pod "f38e2042-5b60-4825-a7b0-1758413fa3ac" (UID: "f38e2042-5b60-4825-a7b0-1758413fa3ac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 21:44:09.919750 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:09.919670 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f38e2042-5b60-4825-a7b0-1758413fa3ac-home" (OuterVolumeSpecName: "home") pod "f38e2042-5b60-4825-a7b0-1758413fa3ac" (UID: "f38e2042-5b60-4825-a7b0-1758413fa3ac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 21:44:09.919750 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:09.919681 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f38e2042-5b60-4825-a7b0-1758413fa3ac-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f38e2042-5b60-4825-a7b0-1758413fa3ac" (UID: "f38e2042-5b60-4825-a7b0-1758413fa3ac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 21:44:09.919860 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:09.919840 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f38e2042-5b60-4825-a7b0-1758413fa3ac-model-cache\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:44:09.919913 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:09.919866 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f38e2042-5b60-4825-a7b0-1758413fa3ac-tmp-dir\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:44:09.919913 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:09.919880 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f38e2042-5b60-4825-a7b0-1758413fa3ac-home\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:44:09.921565 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:09.921538 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f38e2042-5b60-4825-a7b0-1758413fa3ac-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f38e2042-5b60-4825-a7b0-1758413fa3ac" (UID: "f38e2042-5b60-4825-a7b0-1758413fa3ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 21:44:09.921676 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:09.921591 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f38e2042-5b60-4825-a7b0-1758413fa3ac-dshm" (OuterVolumeSpecName: "dshm") pod "f38e2042-5b60-4825-a7b0-1758413fa3ac" (UID: "f38e2042-5b60-4825-a7b0-1758413fa3ac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 21:44:09.921676 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:09.921602 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f38e2042-5b60-4825-a7b0-1758413fa3ac-kube-api-access-fphqq" (OuterVolumeSpecName: "kube-api-access-fphqq") pod "f38e2042-5b60-4825-a7b0-1758413fa3ac" (UID: "f38e2042-5b60-4825-a7b0-1758413fa3ac"). InnerVolumeSpecName "kube-api-access-fphqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 21:44:10.021005 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:10.020977 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f38e2042-5b60-4825-a7b0-1758413fa3ac-dshm\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:44:10.021005 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:10.021002 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f38e2042-5b60-4825-a7b0-1758413fa3ac-tls-certs\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:44:10.021005 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:10.021013 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fphqq\" (UniqueName: \"kubernetes.io/projected/f38e2042-5b60-4825-a7b0-1758413fa3ac-kube-api-access-fphqq\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:44:10.160994 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:10.160892 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:44:10.596188 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:10.596130 2562 generic.go:356] "Generic (PLEG): container finished" podID="f38e2042-5b60-4825-a7b0-1758413fa3ac" containerID="04bcf4b9a691691342e0722469248238238ac34755708388aa9523888c406d9a" exitCode=0 Jul 20 21:44:10.596188 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:10.596186 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2ad633/test-model-2ad633-kserve-6cbcdb9459-ltdm2" Jul 20 21:44:10.596422 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:10.596229 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2ad633/test-model-2ad633-kserve-6cbcdb9459-ltdm2" event={"ID":"f38e2042-5b60-4825-a7b0-1758413fa3ac","Type":"ContainerDied","Data":"04bcf4b9a691691342e0722469248238238ac34755708388aa9523888c406d9a"} Jul 20 21:44:10.596422 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:10.596270 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2ad633/test-model-2ad633-kserve-6cbcdb9459-ltdm2" event={"ID":"f38e2042-5b60-4825-a7b0-1758413fa3ac","Type":"ContainerDied","Data":"014b902fd9b6d7ad851bd6462757159352e6cc0d8ed46599bb9b6e5907fff9c1"} Jul 20 21:44:10.596422 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:10.596290 2562 scope.go:117] "RemoveContainer" containerID="04bcf4b9a691691342e0722469248238238ac34755708388aa9523888c406d9a" Jul 20 21:44:10.605467 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:10.605451 2562 scope.go:117] "RemoveContainer" containerID="04bcf4b9a691691342e0722469248238238ac34755708388aa9523888c406d9a" Jul 20 21:44:10.605749 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:44:10.605732 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04bcf4b9a691691342e0722469248238238ac34755708388aa9523888c406d9a\": container with ID starting with 04bcf4b9a691691342e0722469248238238ac34755708388aa9523888c406d9a not found: ID does not exist" containerID="04bcf4b9a691691342e0722469248238238ac34755708388aa9523888c406d9a" Jul 20 21:44:10.605785 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:10.605757 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04bcf4b9a691691342e0722469248238238ac34755708388aa9523888c406d9a"} err="failed to get container status \"04bcf4b9a691691342e0722469248238238ac34755708388aa9523888c406d9a\": rpc error: code = NotFound desc = could not find container \"04bcf4b9a691691342e0722469248238238ac34755708388aa9523888c406d9a\": container with ID starting with 04bcf4b9a691691342e0722469248238238ac34755708388aa9523888c406d9a not found: ID does not exist" Jul 20 21:44:10.617581 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:10.617559 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2ad633/test-model-2ad633-kserve-6cbcdb9459-ltdm2"] Jul 20 21:44:10.626176 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:10.626149 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-2ad633/test-model-2ad633-kserve-6cbcdb9459-ltdm2"] Jul 20 21:44:11.272022 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:11.271989 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="706b6d30-8899-4a33-bfe0-f5d1324ec215" path="/var/lib/kubelet/pods/706b6d30-8899-4a33-bfe0-f5d1324ec215/volumes" Jul 20 21:44:11.272427 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:11.272417 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f38e2042-5b60-4825-a7b0-1758413fa3ac" path="/var/lib/kubelet/pods/f38e2042-5b60-4825-a7b0-1758413fa3ac/volumes" Jul 20 21:44:16.443953 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:16.443917 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:44:17.092787 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:17.092756 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2ad633-f7qzm"] Jul 20 21:44:17.093186 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:17.093172 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f38e2042-5b60-4825-a7b0-1758413fa3ac" containerName="main" Jul 20 21:44:17.093249 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:17.093187 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f38e2042-5b60-4825-a7b0-1758413fa3ac" containerName="main" Jul 20 21:44:17.093249 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:17.093200 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="706b6d30-8899-4a33-bfe0-f5d1324ec215" containerName="main" Jul 20 21:44:17.093249 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:17.093206 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="706b6d30-8899-4a33-bfe0-f5d1324ec215" containerName="main" Jul 20 21:44:17.093351 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:17.093275 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="706b6d30-8899-4a33-bfe0-f5d1324ec215" containerName="main" Jul 20 21:44:17.093351 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:17.093285 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f38e2042-5b60-4825-a7b0-1758413fa3ac" containerName="main" Jul 20 21:44:17.097535 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:17.097518 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2ad633-f7qzm" Jul 20 21:44:17.101216 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:17.101189 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qr6f9\"" Jul 20 21:44:17.101447 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:17.101430 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 20 21:44:17.107738 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:17.107707 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2ad633-f7qzm"] Jul 20 21:44:17.191525 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:17.191490 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0539a505-4b3e-4c15-adb9-772a2584bc2c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2ad633-f7qzm\" (UID: \"0539a505-4b3e-4c15-adb9-772a2584bc2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2ad633-f7qzm" Jul 20 21:44:17.292486 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:17.292454 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0539a505-4b3e-4c15-adb9-772a2584bc2c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2ad633-f7qzm\" (UID: \"0539a505-4b3e-4c15-adb9-772a2584bc2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2ad633-f7qzm" Jul 20 21:44:17.293074 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:17.293053 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0539a505-4b3e-4c15-adb9-772a2584bc2c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2ad633-f7qzm\" (UID: \"0539a505-4b3e-4c15-adb9-772a2584bc2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2ad633-f7qzm" Jul 20 21:44:17.409757 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:17.409657 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2ad633-f7qzm" Jul 20 21:44:17.554499 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:17.554470 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2ad633-f7qzm"] Jul 20 21:44:17.556892 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:44:17.556865 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0539a505_4b3e_4c15_adb9_772a2584bc2c.slice/crio-2d5fcad6e1e96944cd881e96a0edd810d6e3ae48753ca330d4a1493c4302b758 WatchSource:0}: Error finding container 2d5fcad6e1e96944cd881e96a0edd810d6e3ae48753ca330d4a1493c4302b758: Status 404 returned error can't find the container with id 2d5fcad6e1e96944cd881e96a0edd810d6e3ae48753ca330d4a1493c4302b758 Jul 20 21:44:17.559317 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:17.559295 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 21:44:17.630964 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:17.630929 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2ad633-f7qzm" event={"ID":"0539a505-4b3e-4c15-adb9-772a2584bc2c","Type":"ContainerStarted","Data":"2d5fcad6e1e96944cd881e96a0edd810d6e3ae48753ca330d4a1493c4302b758"} Jul 20 21:44:17.752686 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:17.752598 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:44:21.654541 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:21.654503 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2ad633-f7qzm" event={"ID":"0539a505-4b3e-4c15-adb9-772a2584bc2c","Type":"ContainerStarted","Data":"22a5af851e42405a8ed0508fd7e9f8644ad957703f01572a5a5b4b421183c99f"} Jul 20 21:44:21.673690 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:21.673641 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2ad633-f7qzm" podStartSLOduration=1.386566578 podStartE2EDuration="4.673625147s" totalImagesPullingTime="3.287058569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 21:44:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:44:21.670901891 +0000 UTC m=+2500.996604835" watchObservedRunningTime="2026-07-20 21:44:21.673625147 +0000 UTC m=+2500.999328089" Jul 20 21:44:22.660823 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:22.660785 2562 generic.go:356] "Generic (PLEG): container finished" podID="0539a505-4b3e-4c15-adb9-772a2584bc2c" containerID="22a5af851e42405a8ed0508fd7e9f8644ad957703f01572a5a5b4b421183c99f" exitCode=0 Jul 20 21:44:22.661238 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:22.660864 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2ad633-f7qzm" event={"ID":"0539a505-4b3e-4c15-adb9-772a2584bc2c","Type":"ContainerDied","Data":"22a5af851e42405a8ed0508fd7e9f8644ad957703f01572a5a5b4b421183c99f"} Jul 20 21:44:23.796700 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:23.796672 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2ad633-f7qzm" Jul 20 21:44:23.854599 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:23.854568 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0539a505-4b3e-4c15-adb9-772a2584bc2c-maas-api-service-ca\") pod \"0539a505-4b3e-4c15-adb9-772a2584bc2c\" (UID: \"0539a505-4b3e-4c15-adb9-772a2584bc2c\") " Jul 20 21:44:23.854924 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:23.854903 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0539a505-4b3e-4c15-adb9-772a2584bc2c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0539a505-4b3e-4c15-adb9-772a2584bc2c" (UID: "0539a505-4b3e-4c15-adb9-772a2584bc2c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 21:44:23.955497 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:23.955414 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0539a505-4b3e-4c15-adb9-772a2584bc2c-maas-api-service-ca\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:44:24.671909 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:24.671880 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2ad633-f7qzm" Jul 20 21:44:24.671909 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:24.671894 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2ad633-f7qzm" event={"ID":"0539a505-4b3e-4c15-adb9-772a2584bc2c","Type":"ContainerDied","Data":"2d5fcad6e1e96944cd881e96a0edd810d6e3ae48753ca330d4a1493c4302b758"} Jul 20 21:44:24.672226 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:24.671936 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d5fcad6e1e96944cd881e96a0edd810d6e3ae48753ca330d4a1493c4302b758" Jul 20 21:44:27.648901 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:27.648863 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:44:32.183779 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:32.183744 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2ad633-f7qzm"] Jul 20 21:44:32.186976 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:32.186948 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:44:32.187910 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:32.187887 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2ad633-f7qzm"] Jul 20 21:44:32.580862 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:32.580780 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:44:33.270585 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:33.270551 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0539a505-4b3e-4c15-adb9-772a2584bc2c" path="/var/lib/kubelet/pods/0539a505-4b3e-4c15-adb9-772a2584bc2c/volumes" Jul 20 21:44:38.886353 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:38.886317 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:44:38.984424 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:38.984394 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:44:39.188304 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:39.188221 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:44:39.431796 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:39.431769 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:44:39.842658 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:39.842621 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:44:40.203058 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:40.202972 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:44:40.597633 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:40.597606 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:44:43.600874 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:43.600831 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:44:43.686816 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:43.686783 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d449d6-9849d6b69-prx9g"] Jul 20 21:44:43.687189 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:43.687176 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0539a505-4b3e-4c15-adb9-772a2584bc2c" containerName="revoke-keys" Jul 20 21:44:43.687238 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:43.687190 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0539a505-4b3e-4c15-adb9-772a2584bc2c" containerName="revoke-keys" Jul 20 21:44:43.687278 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:43.687263 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0539a505-4b3e-4c15-adb9-772a2584bc2c" containerName="revoke-keys" Jul 20 21:44:43.695453 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:43.695431 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d449d6-9849d6b69-prx9g" Jul 20 21:44:43.696948 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:43.696916 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:44:43.700744 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:43.700724 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 20 21:44:43.701218 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:43.701022 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-4pmwd\"" Jul 20 21:44:43.701218 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:43.701095 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-d449d6-serving-cert\"" Jul 20 21:44:43.701741 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:43.701724 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 20 21:44:43.713387 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:43.713366 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d449d6-9849d6b69-prx9g"] Jul 20 21:44:43.819503 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:43.819465 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:44:43.846188 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:43.846161 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f03b4b99-3860-4cb4-8c0e-7c34519dc56d-maas-api-tls\") pod \"maas-api-e2e-api-a-d449d6-9849d6b69-prx9g\" (UID: \"f03b4b99-3860-4cb4-8c0e-7c34519dc56d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d449d6-9849d6b69-prx9g" Jul 20 21:44:43.846396 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:43.846218 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdhdc\" (UniqueName: \"kubernetes.io/projected/f03b4b99-3860-4cb4-8c0e-7c34519dc56d-kube-api-access-qdhdc\") pod \"maas-api-e2e-api-a-d449d6-9849d6b69-prx9g\" (UID: \"f03b4b99-3860-4cb4-8c0e-7c34519dc56d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d449d6-9849d6b69-prx9g" Jul 20 21:44:43.846396 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:43.846350 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f03b4b99-3860-4cb4-8c0e-7c34519dc56d-openshift-service-ca\") pod \"maas-api-e2e-api-a-d449d6-9849d6b69-prx9g\" (UID: \"f03b4b99-3860-4cb4-8c0e-7c34519dc56d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d449d6-9849d6b69-prx9g" Jul 20 21:44:43.947530 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:43.947446 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f03b4b99-3860-4cb4-8c0e-7c34519dc56d-maas-api-tls\") pod \"maas-api-e2e-api-a-d449d6-9849d6b69-prx9g\" (UID: \"f03b4b99-3860-4cb4-8c0e-7c34519dc56d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d449d6-9849d6b69-prx9g" Jul 20 21:44:43.947530 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:43.947511 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdhdc\" (UniqueName: \"kubernetes.io/projected/f03b4b99-3860-4cb4-8c0e-7c34519dc56d-kube-api-access-qdhdc\") pod \"maas-api-e2e-api-a-d449d6-9849d6b69-prx9g\" (UID: \"f03b4b99-3860-4cb4-8c0e-7c34519dc56d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d449d6-9849d6b69-prx9g" Jul 20 21:44:43.947730 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:43.947569 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f03b4b99-3860-4cb4-8c0e-7c34519dc56d-openshift-service-ca\") pod \"maas-api-e2e-api-a-d449d6-9849d6b69-prx9g\" (UID: \"f03b4b99-3860-4cb4-8c0e-7c34519dc56d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d449d6-9849d6b69-prx9g" Jul 20 21:44:43.947730 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:44:43.947607 2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-d449d6-serving-cert: secret "maas-api-e2e-api-a-d449d6-serving-cert" not found Jul 20 21:44:43.947730 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:44:43.947697 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f03b4b99-3860-4cb4-8c0e-7c34519dc56d-maas-api-tls podName:f03b4b99-3860-4cb4-8c0e-7c34519dc56d nodeName:}" failed. No retries permitted until 2026-07-20 21:44:44.447675821 +0000 UTC m=+2523.773378753 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f03b4b99-3860-4cb4-8c0e-7c34519dc56d-maas-api-tls") pod "maas-api-e2e-api-a-d449d6-9849d6b69-prx9g" (UID: "f03b4b99-3860-4cb4-8c0e-7c34519dc56d") : secret "maas-api-e2e-api-a-d449d6-serving-cert" not found Jul 20 21:44:43.948262 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:43.948245 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f03b4b99-3860-4cb4-8c0e-7c34519dc56d-openshift-service-ca\") pod \"maas-api-e2e-api-a-d449d6-9849d6b69-prx9g\" (UID: \"f03b4b99-3860-4cb4-8c0e-7c34519dc56d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d449d6-9849d6b69-prx9g" Jul 20 21:44:43.958612 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:43.958594 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdhdc\" (UniqueName: \"kubernetes.io/projected/f03b4b99-3860-4cb4-8c0e-7c34519dc56d-kube-api-access-qdhdc\") pod \"maas-api-e2e-api-a-d449d6-9849d6b69-prx9g\" (UID: \"f03b4b99-3860-4cb4-8c0e-7c34519dc56d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d449d6-9849d6b69-prx9g" Jul 20 21:44:44.079310 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:44.079279 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:44:44.452197 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:44.452129 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f03b4b99-3860-4cb4-8c0e-7c34519dc56d-maas-api-tls\") pod \"maas-api-e2e-api-a-d449d6-9849d6b69-prx9g\" (UID: \"f03b4b99-3860-4cb4-8c0e-7c34519dc56d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d449d6-9849d6b69-prx9g" Jul 20 21:44:44.454803 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:44.454773 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f03b4b99-3860-4cb4-8c0e-7c34519dc56d-maas-api-tls\") pod \"maas-api-e2e-api-a-d449d6-9849d6b69-prx9g\" (UID: \"f03b4b99-3860-4cb4-8c0e-7c34519dc56d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d449d6-9849d6b69-prx9g" Jul 20 21:44:44.462885 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:44.462844 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:44:44.577353 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:44.577318 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:44:44.606557 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:44.606529 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d449d6-9849d6b69-prx9g" Jul 20 21:44:44.752730 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:44.752656 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d449d6-9849d6b69-prx9g"] Jul 20 21:44:44.756316 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:44:44.756292 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf03b4b99_3860_4cb4_8c0e_7c34519dc56d.slice/crio-92330f1779d64581554ffcdee119469073f4eeb599bdedd348412c0945e3b355 WatchSource:0}: Error finding container 92330f1779d64581554ffcdee119469073f4eeb599bdedd348412c0945e3b355: Status 404 returned error can't find the container with id 92330f1779d64581554ffcdee119469073f4eeb599bdedd348412c0945e3b355 Jul 20 21:44:44.769100 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:44.769074 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d449d6-9849d6b69-prx9g" event={"ID":"f03b4b99-3860-4cb4-8c0e-7c34519dc56d","Type":"ContainerStarted","Data":"92330f1779d64581554ffcdee119469073f4eeb599bdedd348412c0945e3b355"} Jul 20 21:44:47.787444 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:47.787407 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d449d6-9849d6b69-prx9g" event={"ID":"f03b4b99-3860-4cb4-8c0e-7c34519dc56d","Type":"ContainerStarted","Data":"06dfbbd3b58d6919e82aa794cda7bc39a606559f2277d85d0ff17b9507dcee85"} Jul 20 21:44:47.787831 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:47.787567 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d449d6-9849d6b69-prx9g" Jul 20 21:44:47.809389 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:47.809322 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d449d6-9849d6b69-prx9g" podStartSLOduration=2.552166789 podStartE2EDuration="4.80930163s" totalImagesPullingTime="2.257134841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 21:44:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:44:47.806660868 +0000 UTC m=+2527.132363812" watchObservedRunningTime="2026-07-20 21:44:47.80930163 +0000 UTC m=+2527.135004575" Jul 20 21:44:48.157683 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:48.157571 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:44:48.656508 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:48.656472 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:44:49.039061 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:49.039011 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:44:49.132920 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:49.132880 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:44:52.105425 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:52.105385 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:44:52.217055 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:52.217022 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:44:52.314369 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:52.314335 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:44:52.414062 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:52.413977 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:44:52.796460 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:52.796428 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:44:52.897405 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:52.897362 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:44:53.797125 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:53.797093 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d449d6-9849d6b69-prx9g" Jul 20 21:44:56.568710 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:44:56.568676 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:45:00.132268 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:00.132231 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29743065-jd5d6"] Jul 20 21:45:00.135824 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:00.135801 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-08ebd1-29743065-b55vj"] Jul 20 21:45:00.135971 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:00.135921 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743065-jd5d6" Jul 20 21:45:00.138721 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:00.138698 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qr6f9\"" Jul 20 21:45:00.138989 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:00.138975 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d449d6-29743065-ptws7"] Jul 20 21:45:00.139116 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:00.139094 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-08ebd1-29743065-b55vj" Jul 20 21:45:00.142095 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:00.142078 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d449d6-29743065-ptws7" Jul 20 21:45:00.147776 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:00.147699 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29743065-jd5d6"] Jul 20 21:45:00.147988 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:00.147909 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743065-jd5d6" Jul 20 21:45:00.150289 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:00.150269 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-08ebd1-29743065-b55vj"] Jul 20 21:45:00.157510 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:00.157489 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-08ebd1-29743065-b55vj" Jul 20 21:45:00.168781 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:00.168683 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d449d6-29743065-ptws7"] Jul 20 21:45:00.195712 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:00.195673 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d449d6-29743065-ptws7" Jul 20 21:45:00.338912 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:00.338881 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-08ebd1-29743065-b55vj"] Jul 20 21:45:00.340272 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:45:00.340247 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff8f191f_2ffc_43e1_97e0_71bb84c3561c.slice/crio-874d85c8bca78f75f10d01e577dfd96e7c48bca6edbb8cd27b0bb469580260e0 WatchSource:0}: Error finding container 874d85c8bca78f75f10d01e577dfd96e7c48bca6edbb8cd27b0bb469580260e0: Status 404 returned error can't find the container with id 874d85c8bca78f75f10d01e577dfd96e7c48bca6edbb8cd27b0bb469580260e0 Jul 20 21:45:00.362879 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:00.362853 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d449d6-29743065-ptws7"] Jul 20 21:45:00.373169 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:45:00.373123 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod867d7a0d_e48b_4fcf_a029_de9ad9635f08.slice/crio-d047c1b90bbfcd827c6c85e4e911fc1c26e1ac0ddc6d7624552d96af33829ea3 WatchSource:0}: Error finding container d047c1b90bbfcd827c6c85e4e911fc1c26e1ac0ddc6d7624552d96af33829ea3: Status 404 returned error can't find the container with id d047c1b90bbfcd827c6c85e4e911fc1c26e1ac0ddc6d7624552d96af33829ea3 Jul 20 21:45:00.515888 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:00.515864 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29743065-jd5d6"] Jul 20 21:45:00.517723 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:45:00.517699 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c72bde9_5a9d_40a4_9745_3d6180216ed3.slice/crio-69fa923f317b9f12f5b6f3249baf202c686f6136cec0c88fd1d4be72df4dab83 WatchSource:0}: Error finding container 69fa923f317b9f12f5b6f3249baf202c686f6136cec0c88fd1d4be72df4dab83: Status 404 returned error can't find the container with id 69fa923f317b9f12f5b6f3249baf202c686f6136cec0c88fd1d4be72df4dab83 Jul 20 21:45:00.857362 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:00.857305 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743065-jd5d6" event={"ID":"7c72bde9-5a9d-40a4-9745-3d6180216ed3","Type":"ContainerStarted","Data":"8246ecb4360668ec796cb57579e99fa59589d37803812cecb0c1f97cad5ebbcb"} Jul 20 21:45:00.857553 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:00.857382 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743065-jd5d6" event={"ID":"7c72bde9-5a9d-40a4-9745-3d6180216ed3","Type":"ContainerStarted","Data":"69fa923f317b9f12f5b6f3249baf202c686f6136cec0c88fd1d4be72df4dab83"} Jul 20 21:45:00.858808 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:00.858781 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d449d6-29743065-ptws7" event={"ID":"867d7a0d-e48b-4fcf-a029-de9ad9635f08","Type":"ContainerStarted","Data":"d1aebf6249bbe4f8c0a2dc7dae6a56ebff00533cb9ece3d640413545c2f7efbe"} Jul 20 21:45:00.858914 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:00.858830 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d449d6-29743065-ptws7" event={"ID":"867d7a0d-e48b-4fcf-a029-de9ad9635f08","Type":"ContainerStarted","Data":"d047c1b90bbfcd827c6c85e4e911fc1c26e1ac0ddc6d7624552d96af33829ea3"} Jul 20 21:45:00.860372 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:00.860343 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-08ebd1-29743065-b55vj" event={"ID":"ff8f191f-2ffc-43e1-97e0-71bb84c3561c","Type":"ContainerStarted","Data":"6cbe4da4252b5c28cef3a9f432f8f4951f2311ab346ff6491571e721dd74ef69"} Jul 20 21:45:00.860458 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:00.860397 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-08ebd1-29743065-b55vj" event={"ID":"ff8f191f-2ffc-43e1-97e0-71bb84c3561c","Type":"ContainerStarted","Data":"874d85c8bca78f75f10d01e577dfd96e7c48bca6edbb8cd27b0bb469580260e0"} Jul 20 21:45:00.875074 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:00.875035 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743065-jd5d6" podStartSLOduration=0.87502066 podStartE2EDuration="875.02066ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:45:00.873892268 +0000 UTC m=+2540.199595212" watchObservedRunningTime="2026-07-20 21:45:00.87502066 +0000 UTC m=+2540.200723602" Jul 20 21:45:00.891440 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:00.891396 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-08ebd1-29743065-b55vj" podStartSLOduration=0.891382178 podStartE2EDuration="891.382178ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:45:00.889811655 +0000 UTC m=+2540.215514597" watchObservedRunningTime="2026-07-20 21:45:00.891382178 +0000 UTC m=+2540.217085121" Jul 20 21:45:00.907344 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:00.907305 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d449d6-29743065-ptws7" podStartSLOduration=0.907290822 podStartE2EDuration="907.290822ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:45:00.906038743 +0000 UTC m=+2540.231741686" watchObservedRunningTime="2026-07-20 21:45:00.907290822 +0000 UTC m=+2540.232993764" Jul 20 21:45:01.866111 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:01.866073 2562 generic.go:356] "Generic (PLEG): container finished" podID="7c72bde9-5a9d-40a4-9745-3d6180216ed3" containerID="8246ecb4360668ec796cb57579e99fa59589d37803812cecb0c1f97cad5ebbcb" exitCode=0 Jul 20 21:45:01.866526 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:01.866169 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743065-jd5d6" event={"ID":"7c72bde9-5a9d-40a4-9745-3d6180216ed3","Type":"ContainerDied","Data":"8246ecb4360668ec796cb57579e99fa59589d37803812cecb0c1f97cad5ebbcb"} Jul 20 21:45:01.867651 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:01.867626 2562 generic.go:356] "Generic (PLEG): container finished" podID="867d7a0d-e48b-4fcf-a029-de9ad9635f08" containerID="d1aebf6249bbe4f8c0a2dc7dae6a56ebff00533cb9ece3d640413545c2f7efbe" exitCode=0 Jul 20 21:45:01.867794 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:01.867708 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d449d6-29743065-ptws7" event={"ID":"867d7a0d-e48b-4fcf-a029-de9ad9635f08","Type":"ContainerDied","Data":"d1aebf6249bbe4f8c0a2dc7dae6a56ebff00533cb9ece3d640413545c2f7efbe"} Jul 20 21:45:01.869054 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:01.869035 2562 generic.go:356] "Generic (PLEG): container finished" podID="ff8f191f-2ffc-43e1-97e0-71bb84c3561c" containerID="6cbe4da4252b5c28cef3a9f432f8f4951f2311ab346ff6491571e721dd74ef69" exitCode=0 Jul 20 21:45:01.869175 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:01.869086 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-08ebd1-29743065-b55vj" event={"ID":"ff8f191f-2ffc-43e1-97e0-71bb84c3561c","Type":"ContainerDied","Data":"6cbe4da4252b5c28cef3a9f432f8f4951f2311ab346ff6491571e721dd74ef69"} Jul 20 21:45:03.029020 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:03.028913 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743065-jd5d6" Jul 20 21:45:03.075530 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:03.075491 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-08ebd1-29743065-b55vj" Jul 20 21:45:03.091090 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:03.091067 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d449d6-29743065-ptws7" Jul 20 21:45:03.880674 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:03.880640 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d449d6-29743065-ptws7" Jul 20 21:45:03.880674 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:03.880656 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d449d6-29743065-ptws7" event={"ID":"867d7a0d-e48b-4fcf-a029-de9ad9635f08","Type":"ContainerDied","Data":"d047c1b90bbfcd827c6c85e4e911fc1c26e1ac0ddc6d7624552d96af33829ea3"} Jul 20 21:45:03.880949 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:03.880706 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d047c1b90bbfcd827c6c85e4e911fc1c26e1ac0ddc6d7624552d96af33829ea3" Jul 20 21:45:03.882008 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:03.881985 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-08ebd1-29743065-b55vj" Jul 20 21:45:03.882155 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:03.882016 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-08ebd1-29743065-b55vj" event={"ID":"ff8f191f-2ffc-43e1-97e0-71bb84c3561c","Type":"ContainerDied","Data":"874d85c8bca78f75f10d01e577dfd96e7c48bca6edbb8cd27b0bb469580260e0"} Jul 20 21:45:03.882155 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:03.882065 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="874d85c8bca78f75f10d01e577dfd96e7c48bca6edbb8cd27b0bb469580260e0" Jul 20 21:45:03.883404 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:03.883388 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743065-jd5d6" Jul 20 21:45:03.883404 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:03.883394 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743065-jd5d6" event={"ID":"7c72bde9-5a9d-40a4-9745-3d6180216ed3","Type":"ContainerDied","Data":"69fa923f317b9f12f5b6f3249baf202c686f6136cec0c88fd1d4be72df4dab83"} Jul 20 21:45:03.883524 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:03.883422 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69fa923f317b9f12f5b6f3249baf202c686f6136cec0c88fd1d4be72df4dab83" Jul 20 21:45:04.755740 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:04.755702 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d449d6-w9znv"] Jul 20 21:45:04.756166 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:04.756093 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff8f191f-2ffc-43e1-97e0-71bb84c3561c" containerName="cleanup" Jul 20 21:45:04.756166 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:04.756103 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff8f191f-2ffc-43e1-97e0-71bb84c3561c" containerName="cleanup" Jul 20 21:45:04.756166 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:04.756131 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="867d7a0d-e48b-4fcf-a029-de9ad9635f08" containerName="cleanup" Jul 20 21:45:04.756166 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:04.756150 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="867d7a0d-e48b-4fcf-a029-de9ad9635f08" containerName="cleanup" Jul 20 21:45:04.756166 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:04.756158 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c72bde9-5a9d-40a4-9745-3d6180216ed3" containerName="cleanup" Jul 20 21:45:04.756166 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:04.756164 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c72bde9-5a9d-40a4-9745-3d6180216ed3" containerName="cleanup" Jul 20 21:45:04.756399 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:04.756218 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="867d7a0d-e48b-4fcf-a029-de9ad9635f08" containerName="cleanup" Jul 20 21:45:04.756399 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:04.756228 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c72bde9-5a9d-40a4-9745-3d6180216ed3" containerName="cleanup" Jul 20 21:45:04.756399 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:04.756235 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff8f191f-2ffc-43e1-97e0-71bb84c3561c" containerName="cleanup" Jul 20 21:45:04.759089 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:04.759072 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d449d6-w9znv" Jul 20 21:45:04.761715 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:04.761698 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qr6f9\"" Jul 20 21:45:04.769647 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:04.769622 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d449d6-w9znv"] Jul 20 21:45:04.848425 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:04.848393 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0407330-cd58-483b-a235-389f00565b10-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d449d6-w9znv\" (UID: \"c0407330-cd58-483b-a235-389f00565b10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d449d6-w9znv" Jul 20 21:45:04.949117 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:04.949083 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0407330-cd58-483b-a235-389f00565b10-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d449d6-w9znv\" (UID: \"c0407330-cd58-483b-a235-389f00565b10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d449d6-w9znv" Jul 20 21:45:04.949770 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:04.949750 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0407330-cd58-483b-a235-389f00565b10-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d449d6-w9znv\" (UID: \"c0407330-cd58-483b-a235-389f00565b10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d449d6-w9znv" Jul 20 21:45:05.070799 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:05.070760 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d449d6-w9znv" Jul 20 21:45:05.206776 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:05.206748 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d449d6-w9znv"] Jul 20 21:45:05.209087 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:45:05.209064 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0407330_cd58_483b_a235_389f00565b10.slice/crio-1c974201c4e288942ac5020bc8512bc168998b412f6ced46aad970f55568e00f WatchSource:0}: Error finding container 1c974201c4e288942ac5020bc8512bc168998b412f6ced46aad970f55568e00f: Status 404 returned error can't find the container with id 1c974201c4e288942ac5020bc8512bc168998b412f6ced46aad970f55568e00f Jul 20 21:45:05.406913 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:05.406880 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:45:05.505723 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:05.505685 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:45:05.897658 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:05.897627 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d449d6-w9znv" event={"ID":"c0407330-cd58-483b-a235-389f00565b10","Type":"ContainerStarted","Data":"290f2d68dcf3608a730cd8b2b0ff8797fc67474c93a383bffd38e65160d0f937"} Jul 20 21:45:05.898035 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:05.897684 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d449d6-w9znv" event={"ID":"c0407330-cd58-483b-a235-389f00565b10","Type":"ContainerStarted","Data":"1c974201c4e288942ac5020bc8512bc168998b412f6ced46aad970f55568e00f"} Jul 20 21:45:05.914840 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:05.914765 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d449d6-w9znv" podStartSLOduration=1.914743439 podStartE2EDuration="1.914743439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 21:45:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:45:05.912122109 +0000 UTC m=+2545.237825063" watchObservedRunningTime="2026-07-20 21:45:05.914743439 +0000 UTC m=+2545.240446384" Jul 20 21:45:06.904519 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:06.904483 2562 generic.go:356] "Generic (PLEG): container finished" podID="c0407330-cd58-483b-a235-389f00565b10" containerID="290f2d68dcf3608a730cd8b2b0ff8797fc67474c93a383bffd38e65160d0f937" exitCode=0 Jul 20 21:45:06.904912 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:06.904571 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d449d6-w9znv" event={"ID":"c0407330-cd58-483b-a235-389f00565b10","Type":"ContainerDied","Data":"290f2d68dcf3608a730cd8b2b0ff8797fc67474c93a383bffd38e65160d0f937"} Jul 20 21:45:08.037699 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:08.037676 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d449d6-w9znv" Jul 20 21:45:08.077408 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:08.077377 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0407330-cd58-483b-a235-389f00565b10-maas-api-service-ca\") pod \"c0407330-cd58-483b-a235-389f00565b10\" (UID: \"c0407330-cd58-483b-a235-389f00565b10\") " Jul 20 21:45:08.077719 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:08.077688 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0407330-cd58-483b-a235-389f00565b10-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c0407330-cd58-483b-a235-389f00565b10" (UID: "c0407330-cd58-483b-a235-389f00565b10"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 21:45:08.179082 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:08.178997 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0407330-cd58-483b-a235-389f00565b10-maas-api-service-ca\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:45:08.916194 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:08.916161 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d449d6-w9znv" Jul 20 21:45:08.916194 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:08.916173 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d449d6-w9znv" event={"ID":"c0407330-cd58-483b-a235-389f00565b10","Type":"ContainerDied","Data":"1c974201c4e288942ac5020bc8512bc168998b412f6ced46aad970f55568e00f"} Jul 20 21:45:08.916407 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:08.916225 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c974201c4e288942ac5020bc8512bc168998b412f6ced46aad970f55568e00f" Jul 20 21:45:13.677421 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:13.677385 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:45:13.798905 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:13.798872 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:45:15.514946 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:15.514916 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:45:15.612311 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:15.612285 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:45:20.793619 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:20.793576 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d449d6-9849d6b69-prx9g"] Jul 20 21:45:20.794095 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:20.793838 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d449d6-9849d6b69-prx9g" podUID="f03b4b99-3860-4cb4-8c0e-7c34519dc56d" containerName="maas-api" containerID="cri-o://06dfbbd3b58d6919e82aa794cda7bc39a606559f2277d85d0ff17b9507dcee85" gracePeriod=30 Jul 20 21:45:20.810673 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:20.810647 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d449d6-29743065-ptws7"] Jul 20 21:45:20.816253 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:20.816222 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:45:20.817864 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:20.817838 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-d449d6-29743065-ptws7"] Jul 20 21:45:20.977672 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:20.977643 2562 generic.go:356] "Generic (PLEG): container finished" podID="f03b4b99-3860-4cb4-8c0e-7c34519dc56d" containerID="06dfbbd3b58d6919e82aa794cda7bc39a606559f2277d85d0ff17b9507dcee85" exitCode=0 Jul 20 21:45:20.977827 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:20.977701 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d449d6-9849d6b69-prx9g" event={"ID":"f03b4b99-3860-4cb4-8c0e-7c34519dc56d","Type":"ContainerDied","Data":"06dfbbd3b58d6919e82aa794cda7bc39a606559f2277d85d0ff17b9507dcee85"} Jul 20 21:45:21.037553 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:21.037529 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d449d6-9849d6b69-prx9g" Jul 20 21:45:21.100854 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:21.100758 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f03b4b99-3860-4cb4-8c0e-7c34519dc56d-openshift-service-ca\") pod \"f03b4b99-3860-4cb4-8c0e-7c34519dc56d\" (UID: \"f03b4b99-3860-4cb4-8c0e-7c34519dc56d\") " Jul 20 21:45:21.100854 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:21.100808 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdhdc\" (UniqueName: \"kubernetes.io/projected/f03b4b99-3860-4cb4-8c0e-7c34519dc56d-kube-api-access-qdhdc\") pod \"f03b4b99-3860-4cb4-8c0e-7c34519dc56d\" (UID: \"f03b4b99-3860-4cb4-8c0e-7c34519dc56d\") " Jul 20 21:45:21.101064 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:21.100901 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f03b4b99-3860-4cb4-8c0e-7c34519dc56d-maas-api-tls\") pod \"f03b4b99-3860-4cb4-8c0e-7c34519dc56d\" (UID: \"f03b4b99-3860-4cb4-8c0e-7c34519dc56d\") " Jul 20 21:45:21.101151 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:21.101109 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f03b4b99-3860-4cb4-8c0e-7c34519dc56d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f03b4b99-3860-4cb4-8c0e-7c34519dc56d" (UID: "f03b4b99-3860-4cb4-8c0e-7c34519dc56d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 21:45:21.101251 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:21.101237 2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f03b4b99-3860-4cb4-8c0e-7c34519dc56d-openshift-service-ca\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:45:21.103042 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:21.103025 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f03b4b99-3860-4cb4-8c0e-7c34519dc56d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f03b4b99-3860-4cb4-8c0e-7c34519dc56d" (UID: "f03b4b99-3860-4cb4-8c0e-7c34519dc56d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 21:45:21.103128 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:21.103108 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f03b4b99-3860-4cb4-8c0e-7c34519dc56d-kube-api-access-qdhdc" (OuterVolumeSpecName: "kube-api-access-qdhdc") pod "f03b4b99-3860-4cb4-8c0e-7c34519dc56d" (UID: "f03b4b99-3860-4cb4-8c0e-7c34519dc56d"). InnerVolumeSpecName "kube-api-access-qdhdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 21:45:21.202655 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:21.202614 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f03b4b99-3860-4cb4-8c0e-7c34519dc56d-maas-api-tls\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:45:21.202655 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:21.202650 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdhdc\" (UniqueName: \"kubernetes.io/projected/f03b4b99-3860-4cb4-8c0e-7c34519dc56d-kube-api-access-qdhdc\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:45:21.269996 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:21.269964 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="867d7a0d-e48b-4fcf-a029-de9ad9635f08" path="/var/lib/kubelet/pods/867d7a0d-e48b-4fcf-a029-de9ad9635f08/volumes" Jul 20 21:45:21.984320 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:21.984278 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d449d6-9849d6b69-prx9g" event={"ID":"f03b4b99-3860-4cb4-8c0e-7c34519dc56d","Type":"ContainerDied","Data":"92330f1779d64581554ffcdee119469073f4eeb599bdedd348412c0945e3b355"} Jul 20 21:45:21.984748 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:21.984346 2562 scope.go:117] "RemoveContainer" containerID="06dfbbd3b58d6919e82aa794cda7bc39a606559f2277d85d0ff17b9507dcee85" Jul 20 21:45:21.984748 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:21.984291 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d449d6-9849d6b69-prx9g" Jul 20 21:45:22.002238 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:22.002210 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d449d6-9849d6b69-prx9g"] Jul 20 21:45:22.006106 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:22.006084 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d449d6-9849d6b69-prx9g"] Jul 20 21:45:23.270680 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:23.270645 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f03b4b99-3860-4cb4-8c0e-7c34519dc56d" path="/var/lib/kubelet/pods/f03b4b99-3860-4cb4-8c0e-7c34519dc56d/volumes" Jul 20 21:45:24.848577 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:24.848530 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d449d6-w9znv"] Jul 20 21:45:24.851354 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:24.851331 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d449d6-w9znv"] Jul 20 21:45:24.925449 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:24.925407 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:45:25.269654 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:25.269622 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0407330-cd58-483b-a235-389f00565b10" path="/var/lib/kubelet/pods/c0407330-cd58-483b-a235-389f00565b10/volumes" Jul 20 21:45:25.332926 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:25.332891 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:45:31.110608 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:31.110574 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:45:31.211217 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:31.211183 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:45:31.422575 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:31.422472 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:45:31.679563 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:31.679478 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:45:31.806432 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:31.806399 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08ebd1-rrp4j"] Jul 20 21:45:31.806827 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:31.806811 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f03b4b99-3860-4cb4-8c0e-7c34519dc56d" containerName="maas-api" Jul 20 21:45:31.806827 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:31.806826 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f03b4b99-3860-4cb4-8c0e-7c34519dc56d" containerName="maas-api" Jul 20 21:45:31.806958 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:31.806854 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0407330-cd58-483b-a235-389f00565b10" containerName="revoke-keys" Jul 20 21:45:31.806958 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:31.806860 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0407330-cd58-483b-a235-389f00565b10" containerName="revoke-keys" Jul 20 21:45:31.806958 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:31.806917 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0407330-cd58-483b-a235-389f00565b10" containerName="revoke-keys" Jul 20 21:45:31.806958 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:31.806928 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f03b4b99-3860-4cb4-8c0e-7c34519dc56d" containerName="maas-api" Jul 20 21:45:31.811200 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:31.811172 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08ebd1-rrp4j" Jul 20 21:45:31.813679 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:31.813659 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 20 21:45:31.813796 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:31.813659 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qr6f9\"" Jul 20 21:45:31.817758 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:31.817736 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08ebd1-rrp4j"] Jul 20 21:45:31.906202 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:31.906161 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/baeee362-b2ca-43c8-8fad-feeb6e09528e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-08ebd1-rrp4j\" (UID: \"baeee362-b2ca-43c8-8fad-feeb6e09528e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08ebd1-rrp4j" Jul 20 21:45:32.007090 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:32.007008 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/baeee362-b2ca-43c8-8fad-feeb6e09528e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-08ebd1-rrp4j\" (UID: \"baeee362-b2ca-43c8-8fad-feeb6e09528e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08ebd1-rrp4j" Jul 20 21:45:32.007609 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:32.007590 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/baeee362-b2ca-43c8-8fad-feeb6e09528e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-08ebd1-rrp4j\" (UID: \"baeee362-b2ca-43c8-8fad-feeb6e09528e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08ebd1-rrp4j" Jul 20 21:45:32.122789 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:32.122757 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08ebd1-rrp4j" Jul 20 21:45:32.260197 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:32.260171 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08ebd1-rrp4j"] Jul 20 21:45:32.261902 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:45:32.261870 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaeee362_b2ca_43c8_8fad_feeb6e09528e.slice/crio-73d4e23ffc3acc029a9f00ebceb2f576e90bb332b628e900e2e2a1fd6f75c237 WatchSource:0}: Error finding container 73d4e23ffc3acc029a9f00ebceb2f576e90bb332b628e900e2e2a1fd6f75c237: Status 404 returned error can't find the container with id 73d4e23ffc3acc029a9f00ebceb2f576e90bb332b628e900e2e2a1fd6f75c237 Jul 20 21:45:32.399563 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:32.399531 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:45:32.498304 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:32.498277 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:45:33.041778 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:33.041741 2562 generic.go:356] "Generic (PLEG): container finished" podID="baeee362-b2ca-43c8-8fad-feeb6e09528e" containerID="51c25a5840fac7f57894fe00ce5d47e4ad579593ec2146f9d9b25738f8537386" exitCode=0 Jul 20 21:45:33.041961 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:33.041827 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08ebd1-rrp4j" event={"ID":"baeee362-b2ca-43c8-8fad-feeb6e09528e","Type":"ContainerDied","Data":"51c25a5840fac7f57894fe00ce5d47e4ad579593ec2146f9d9b25738f8537386"} Jul 20 21:45:33.041961 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:33.041885 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08ebd1-rrp4j" event={"ID":"baeee362-b2ca-43c8-8fad-feeb6e09528e","Type":"ContainerStarted","Data":"73d4e23ffc3acc029a9f00ebceb2f576e90bb332b628e900e2e2a1fd6f75c237"} Jul 20 21:45:34.183878 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:34.183853 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08ebd1-rrp4j" Jul 20 21:45:34.327951 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:34.327864 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/baeee362-b2ca-43c8-8fad-feeb6e09528e-maas-api-service-ca\") pod \"baeee362-b2ca-43c8-8fad-feeb6e09528e\" (UID: \"baeee362-b2ca-43c8-8fad-feeb6e09528e\") " Jul 20 21:45:34.328246 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:34.328225 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baeee362-b2ca-43c8-8fad-feeb6e09528e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "baeee362-b2ca-43c8-8fad-feeb6e09528e" (UID: "baeee362-b2ca-43c8-8fad-feeb6e09528e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 21:45:34.429630 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:34.429601 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/baeee362-b2ca-43c8-8fad-feeb6e09528e-maas-api-service-ca\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:45:35.056046 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:35.056014 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08ebd1-rrp4j" event={"ID":"baeee362-b2ca-43c8-8fad-feeb6e09528e","Type":"ContainerDied","Data":"73d4e23ffc3acc029a9f00ebceb2f576e90bb332b628e900e2e2a1fd6f75c237"} Jul 20 21:45:35.056246 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:35.056053 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08ebd1-rrp4j" Jul 20 21:45:35.056246 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:35.056063 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73d4e23ffc3acc029a9f00ebceb2f576e90bb332b628e900e2e2a1fd6f75c237" Jul 20 21:45:46.023040 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:46.023006 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:45:46.122421 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:46.122386 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:45:47.343184 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:47.343149 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-08ebd1-29743065-b55vj"] Jul 20 21:45:47.346463 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:47.346435 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-08ebd1-29743065-b55vj"] Jul 20 21:45:47.347130 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:47.347108 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:45:49.270703 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:49.270670 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff8f191f-2ffc-43e1-97e0-71bb84c3561c" path="/var/lib/kubelet/pods/ff8f191f-2ffc-43e1-97e0-71bb84c3561c/volumes" Jul 20 21:45:51.903805 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:51.903769 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:45:51.918077 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:51.918043 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08ebd1-rrp4j"] Jul 20 21:45:51.920299 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:51.920268 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-08ebd1-rrp4j"] Jul 20 21:45:52.384191 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:52.384112 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:45:53.270640 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:53.270610 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="baeee362-b2ca-43c8-8fad-feeb6e09528e" path="/var/lib/kubelet/pods/baeee362-b2ca-43c8-8fad-feeb6e09528e/volumes" Jul 20 21:45:58.326867 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:58.326819 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:45:58.390836 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:58.390806 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:45:58.650934 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:58.650839 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:45:58.879447 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:58.879408 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:45:59.263869 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:59.263832 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:45:59.595849 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:59.595772 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:45:59.906181 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:45:59.906062 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:46:03.234249 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:03.234165 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:46:03.373941 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:03.373905 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-196c99-cf748797d-nkzxw"] Jul 20 21:46:03.374328 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:03.374314 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baeee362-b2ca-43c8-8fad-feeb6e09528e" containerName="revoke-keys" Jul 20 21:46:03.374376 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:03.374329 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="baeee362-b2ca-43c8-8fad-feeb6e09528e" containerName="revoke-keys" Jul 20 21:46:03.374413 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:03.374406 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="baeee362-b2ca-43c8-8fad-feeb6e09528e" containerName="revoke-keys" Jul 20 21:46:03.377700 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:03.377682 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-196c99-cf748797d-nkzxw" Jul 20 21:46:03.380264 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:03.380240 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 20 21:46:03.380412 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:03.380328 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-196c99-serving-cert\"" Jul 20 21:46:03.380412 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:03.380347 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-4pmwd\"" Jul 20 21:46:03.381342 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:03.381324 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 20 21:46:03.396283 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:03.396254 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-196c99-cf748797d-nkzxw"] Jul 20 21:46:03.421360 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:03.421327 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:46:03.502860 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:03.502756 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rstjr\" (UniqueName: \"kubernetes.io/projected/6703a40c-37a5-450c-8a1f-3c181ab5c20d-kube-api-access-rstjr\") pod \"maas-api-e2e-shared-a-196c99-cf748797d-nkzxw\" (UID: \"6703a40c-37a5-450c-8a1f-3c181ab5c20d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-196c99-cf748797d-nkzxw" Jul 20 21:46:03.502860 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:03.502846 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6703a40c-37a5-450c-8a1f-3c181ab5c20d-maas-api-tls\") pod \"maas-api-e2e-shared-a-196c99-cf748797d-nkzxw\" (UID: \"6703a40c-37a5-450c-8a1f-3c181ab5c20d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-196c99-cf748797d-nkzxw" Jul 20 21:46:03.503072 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:03.502883 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6703a40c-37a5-450c-8a1f-3c181ab5c20d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-196c99-cf748797d-nkzxw\" (UID: \"6703a40c-37a5-450c-8a1f-3c181ab5c20d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-196c99-cf748797d-nkzxw" Jul 20 21:46:03.555389 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:03.555352 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:46:03.604359 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:03.604319 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6703a40c-37a5-450c-8a1f-3c181ab5c20d-maas-api-tls\") pod \"maas-api-e2e-shared-a-196c99-cf748797d-nkzxw\" (UID: \"6703a40c-37a5-450c-8a1f-3c181ab5c20d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-196c99-cf748797d-nkzxw" Jul 20 21:46:03.604359 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:03.604366 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6703a40c-37a5-450c-8a1f-3c181ab5c20d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-196c99-cf748797d-nkzxw\" (UID: \"6703a40c-37a5-450c-8a1f-3c181ab5c20d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-196c99-cf748797d-nkzxw" Jul 20 21:46:03.604587 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:03.604422 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rstjr\" (UniqueName: \"kubernetes.io/projected/6703a40c-37a5-450c-8a1f-3c181ab5c20d-kube-api-access-rstjr\") pod \"maas-api-e2e-shared-a-196c99-cf748797d-nkzxw\" (UID: \"6703a40c-37a5-450c-8a1f-3c181ab5c20d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-196c99-cf748797d-nkzxw" Jul 20 21:46:03.605091 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:03.605069 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6703a40c-37a5-450c-8a1f-3c181ab5c20d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-196c99-cf748797d-nkzxw\" (UID: \"6703a40c-37a5-450c-8a1f-3c181ab5c20d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-196c99-cf748797d-nkzxw" Jul 20 21:46:03.606741 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:03.606723 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6703a40c-37a5-450c-8a1f-3c181ab5c20d-maas-api-tls\") pod \"maas-api-e2e-shared-a-196c99-cf748797d-nkzxw\" (UID: \"6703a40c-37a5-450c-8a1f-3c181ab5c20d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-196c99-cf748797d-nkzxw" Jul 20 21:46:03.623879 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:03.623855 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rstjr\" (UniqueName: \"kubernetes.io/projected/6703a40c-37a5-450c-8a1f-3c181ab5c20d-kube-api-access-rstjr\") pod \"maas-api-e2e-shared-a-196c99-cf748797d-nkzxw\" (UID: \"6703a40c-37a5-450c-8a1f-3c181ab5c20d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-196c99-cf748797d-nkzxw" Jul 20 21:46:03.688386 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:03.688347 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-196c99-cf748797d-nkzxw" Jul 20 21:46:03.699336 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:03.699300 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:46:03.839790 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:03.839709 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-196c99-cf748797d-nkzxw"] Jul 20 21:46:03.842862 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:46:03.842835 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6703a40c_37a5_450c_8a1f_3c181ab5c20d.slice/crio-6f13b80637636b08428585f847965e45abc7d9b969d335aa7a4f62062fab8dea WatchSource:0}: Error finding container 6f13b80637636b08428585f847965e45abc7d9b969d335aa7a4f62062fab8dea: Status 404 returned error can't find the container with id 6f13b80637636b08428585f847965e45abc7d9b969d335aa7a4f62062fab8dea Jul 20 21:46:04.201319 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:04.201236 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-196c99-cf748797d-nkzxw" event={"ID":"6703a40c-37a5-450c-8a1f-3c181ab5c20d","Type":"ContainerStarted","Data":"6f13b80637636b08428585f847965e45abc7d9b969d335aa7a4f62062fab8dea"} Jul 20 21:46:04.281210 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:04.281179 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:46:04.429184 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:04.429120 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:46:05.209034 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:05.208999 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-196c99-cf748797d-nkzxw" event={"ID":"6703a40c-37a5-450c-8a1f-3c181ab5c20d","Type":"ContainerStarted","Data":"988ab7b168a8ddc4d4faca326500350baa36e2f51c7471248558dbb9bc0f6f5a"} Jul 20 21:46:05.209235 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:05.209179 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-196c99-cf748797d-nkzxw" Jul 20 21:46:05.231325 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:05.231267 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-196c99-cf748797d-nkzxw" podStartSLOduration=1.822236456 podStartE2EDuration="2.231250281s" totalImagesPullingTime="409.013825ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 21:46:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:46:05.228777697 +0000 UTC m=+2604.554480639" watchObservedRunningTime="2026-07-20 21:46:05.231250281 +0000 UTC m=+2604.556953238" Jul 20 21:46:07.788056 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:07.788020 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:46:08.144444 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:08.144350 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:46:08.535727 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:08.535697 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:46:08.825515 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:08.825437 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:46:11.189835 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:11.189801 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:46:11.219411 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:11.219386 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-196c99-cf748797d-nkzxw" Jul 20 21:46:11.273451 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:11.273422 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-046c73-68596dc579-w8hpn"] Jul 20 21:46:11.277245 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:11.277222 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-046c73-68596dc579-w8hpn" Jul 20 21:46:11.279866 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:11.279844 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-046c73-serving-cert\"" Jul 20 21:46:11.283359 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:11.283335 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:46:11.288584 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:11.288561 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-046c73-68596dc579-w8hpn"] Jul 20 21:46:11.359613 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:11.359574 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:46:11.379771 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:11.379737 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/732efb04-60db-47c0-838e-5981809ad489-openshift-service-ca\") pod \"maas-api-e2e-shared-b-046c73-68596dc579-w8hpn\" (UID: \"732efb04-60db-47c0-838e-5981809ad489\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-046c73-68596dc579-w8hpn" Jul 20 21:46:11.379960 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:11.379930 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/732efb04-60db-47c0-838e-5981809ad489-maas-api-tls\") pod \"maas-api-e2e-shared-b-046c73-68596dc579-w8hpn\" (UID: \"732efb04-60db-47c0-838e-5981809ad489\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-046c73-68596dc579-w8hpn" Jul 20 21:46:11.380109 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:11.379986 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w2c5\" (UniqueName: \"kubernetes.io/projected/732efb04-60db-47c0-838e-5981809ad489-kube-api-access-8w2c5\") pod \"maas-api-e2e-shared-b-046c73-68596dc579-w8hpn\" (UID: \"732efb04-60db-47c0-838e-5981809ad489\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-046c73-68596dc579-w8hpn" Jul 20 21:46:11.480954 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:11.480865 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/732efb04-60db-47c0-838e-5981809ad489-maas-api-tls\") pod \"maas-api-e2e-shared-b-046c73-68596dc579-w8hpn\" (UID: \"732efb04-60db-47c0-838e-5981809ad489\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-046c73-68596dc579-w8hpn" Jul 20 21:46:11.480954 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:11.480908 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w2c5\" (UniqueName: \"kubernetes.io/projected/732efb04-60db-47c0-838e-5981809ad489-kube-api-access-8w2c5\") pod \"maas-api-e2e-shared-b-046c73-68596dc579-w8hpn\" (UID: \"732efb04-60db-47c0-838e-5981809ad489\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-046c73-68596dc579-w8hpn" Jul 20 21:46:11.480954 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:11.480944 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/732efb04-60db-47c0-838e-5981809ad489-openshift-service-ca\") pod \"maas-api-e2e-shared-b-046c73-68596dc579-w8hpn\" (UID: \"732efb04-60db-47c0-838e-5981809ad489\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-046c73-68596dc579-w8hpn" Jul 20 21:46:11.481315 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:46:11.481018 2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-046c73-serving-cert: secret "maas-api-e2e-shared-b-046c73-serving-cert" not found Jul 20 21:46:11.481315 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:46:11.481102 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/732efb04-60db-47c0-838e-5981809ad489-maas-api-tls podName:732efb04-60db-47c0-838e-5981809ad489 nodeName:}" failed. No retries permitted until 2026-07-20 21:46:11.981083631 +0000 UTC m=+2611.306786565 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/732efb04-60db-47c0-838e-5981809ad489-maas-api-tls") pod "maas-api-e2e-shared-b-046c73-68596dc579-w8hpn" (UID: "732efb04-60db-47c0-838e-5981809ad489") : secret "maas-api-e2e-shared-b-046c73-serving-cert" not found Jul 20 21:46:11.481605 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:11.481586 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/732efb04-60db-47c0-838e-5981809ad489-openshift-service-ca\") pod \"maas-api-e2e-shared-b-046c73-68596dc579-w8hpn\" (UID: \"732efb04-60db-47c0-838e-5981809ad489\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-046c73-68596dc579-w8hpn" Jul 20 21:46:11.493925 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:11.493902 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w2c5\" (UniqueName: \"kubernetes.io/projected/732efb04-60db-47c0-838e-5981809ad489-kube-api-access-8w2c5\") pod \"maas-api-e2e-shared-b-046c73-68596dc579-w8hpn\" (UID: \"732efb04-60db-47c0-838e-5981809ad489\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-046c73-68596dc579-w8hpn" Jul 20 21:46:11.765109 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:11.765076 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:46:11.870784 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:11.870751 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:46:11.969496 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:11.969462 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:46:11.985620 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:11.985584 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/732efb04-60db-47c0-838e-5981809ad489-maas-api-tls\") pod \"maas-api-e2e-shared-b-046c73-68596dc579-w8hpn\" (UID: \"732efb04-60db-47c0-838e-5981809ad489\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-046c73-68596dc579-w8hpn" Jul 20 21:46:11.987994 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:11.987970 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/732efb04-60db-47c0-838e-5981809ad489-maas-api-tls\") pod \"maas-api-e2e-shared-b-046c73-68596dc579-w8hpn\" (UID: \"732efb04-60db-47c0-838e-5981809ad489\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-046c73-68596dc579-w8hpn" Jul 20 21:46:12.190869 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:12.190789 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-046c73-68596dc579-w8hpn" Jul 20 21:46:12.540667 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:12.540635 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-046c73-68596dc579-w8hpn"] Jul 20 21:46:12.541906 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:46:12.541883 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod732efb04_60db_47c0_838e_5981809ad489.slice/crio-65fd47ce789a6b8b3784f71297c38acfa15d113e8d0518428df9d81fd0a178fc WatchSource:0}: Error finding container 65fd47ce789a6b8b3784f71297c38acfa15d113e8d0518428df9d81fd0a178fc: Status 404 returned error can't find the container with id 65fd47ce789a6b8b3784f71297c38acfa15d113e8d0518428df9d81fd0a178fc Jul 20 21:46:13.256421 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:13.256335 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-046c73-68596dc579-w8hpn" event={"ID":"732efb04-60db-47c0-838e-5981809ad489","Type":"ContainerStarted","Data":"1d33432d9c059b54ca0f6093af6afffddf3dbd070fcca4dfadd4a0a6c80308a7"} Jul 20 21:46:13.256421 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:13.256388 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-046c73-68596dc579-w8hpn" event={"ID":"732efb04-60db-47c0-838e-5981809ad489","Type":"ContainerStarted","Data":"65fd47ce789a6b8b3784f71297c38acfa15d113e8d0518428df9d81fd0a178fc"} Jul 20 21:46:13.256822 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:13.256438 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-046c73-68596dc579-w8hpn" Jul 20 21:46:13.281403 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:13.281353 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-046c73-68596dc579-w8hpn" podStartSLOduration=1.9883637570000001 podStartE2EDuration="2.281339847s" totalImagesPullingTime="292.97609ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 21:46:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:46:13.278843042 +0000 UTC m=+2612.604545984" watchObservedRunningTime="2026-07-20 21:46:13.281339847 +0000 UTC m=+2612.607042789" Jul 20 21:46:15.751220 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:15.751185 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:46:15.849865 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:15.849830 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:46:19.269949 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:19.269920 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-046c73-68596dc579-w8hpn" Jul 20 21:46:24.350933 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:24.350903 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:46:26.223483 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:26.223450 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:46:26.324089 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:26.324053 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:46:32.772824 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:32.772789 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:46:32.880286 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:32.880248 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:46:36.048914 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:36.048872 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:46:36.149539 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:36.149501 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:46:42.789540 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:42.789502 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-046c73/auth-test-model-046c73-kserve-858d9d9d87-46dtw"] Jul 20 21:46:42.793317 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:42.793299 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-046c73/auth-test-model-046c73-kserve-858d9d9d87-46dtw" Jul 20 21:46:42.796153 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:42.796115 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-046c73\"/\"openshift-service-ca.crt\"" Jul 20 21:46:42.796963 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:42.796943 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-046c73\"/\"default-dockercfg-rwqs9\"" Jul 20 21:46:42.797054 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:42.796950 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-046c73\"/\"auth-test-model-046c73-kserve-self-signed-certs\"" Jul 20 21:46:42.797105 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:42.796987 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-046c73\"/\"kube-root-ca.crt\"" Jul 20 21:46:42.802927 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:42.802903 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-046c73/auth-test-model-046c73-kserve-858d9d9d87-46dtw"] Jul 20 21:46:42.809843 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:42.809821 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:46:42.881821 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:42.881791 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbcn8\" (UniqueName: \"kubernetes.io/projected/4ef46531-e365-460e-addf-02999d90790d-kube-api-access-lbcn8\") pod \"auth-test-model-046c73-kserve-858d9d9d87-46dtw\" (UID: \"4ef46531-e365-460e-addf-02999d90790d\") " pod="ai-tenant-e2e-shared-b-046c73/auth-test-model-046c73-kserve-858d9d9d87-46dtw" Jul 20 21:46:42.881947 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:42.881879 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ef46531-e365-460e-addf-02999d90790d-tmp-dir\") pod \"auth-test-model-046c73-kserve-858d9d9d87-46dtw\" (UID: \"4ef46531-e365-460e-addf-02999d90790d\") " pod="ai-tenant-e2e-shared-b-046c73/auth-test-model-046c73-kserve-858d9d9d87-46dtw" Jul 20 21:46:42.881947 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:42.881912 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ef46531-e365-460e-addf-02999d90790d-tls-certs\") pod \"auth-test-model-046c73-kserve-858d9d9d87-46dtw\" (UID: \"4ef46531-e365-460e-addf-02999d90790d\") " pod="ai-tenant-e2e-shared-b-046c73/auth-test-model-046c73-kserve-858d9d9d87-46dtw" Jul 20 21:46:42.881947 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:42.881938 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ef46531-e365-460e-addf-02999d90790d-home\") pod \"auth-test-model-046c73-kserve-858d9d9d87-46dtw\" (UID: \"4ef46531-e365-460e-addf-02999d90790d\") " pod="ai-tenant-e2e-shared-b-046c73/auth-test-model-046c73-kserve-858d9d9d87-46dtw" Jul 20 21:46:42.882049 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:42.882022 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ef46531-e365-460e-addf-02999d90790d-dshm\") pod \"auth-test-model-046c73-kserve-858d9d9d87-46dtw\" (UID: \"4ef46531-e365-460e-addf-02999d90790d\") " pod="ai-tenant-e2e-shared-b-046c73/auth-test-model-046c73-kserve-858d9d9d87-46dtw" Jul 20 21:46:42.882088 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:42.882065 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ef46531-e365-460e-addf-02999d90790d-model-cache\") pod \"auth-test-model-046c73-kserve-858d9d9d87-46dtw\" (UID: \"4ef46531-e365-460e-addf-02999d90790d\") " pod="ai-tenant-e2e-shared-b-046c73/auth-test-model-046c73-kserve-858d9d9d87-46dtw" Jul 20 21:46:42.982920 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:42.982887 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ef46531-e365-460e-addf-02999d90790d-dshm\") pod \"auth-test-model-046c73-kserve-858d9d9d87-46dtw\" (UID: \"4ef46531-e365-460e-addf-02999d90790d\") " pod="ai-tenant-e2e-shared-b-046c73/auth-test-model-046c73-kserve-858d9d9d87-46dtw" Jul 20 21:46:42.982920 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:42.982924 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ef46531-e365-460e-addf-02999d90790d-model-cache\") pod \"auth-test-model-046c73-kserve-858d9d9d87-46dtw\" (UID: \"4ef46531-e365-460e-addf-02999d90790d\") " pod="ai-tenant-e2e-shared-b-046c73/auth-test-model-046c73-kserve-858d9d9d87-46dtw" Jul 20 21:46:42.983117 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:42.983015 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbcn8\" (UniqueName: \"kubernetes.io/projected/4ef46531-e365-460e-addf-02999d90790d-kube-api-access-lbcn8\") pod \"auth-test-model-046c73-kserve-858d9d9d87-46dtw\" (UID: \"4ef46531-e365-460e-addf-02999d90790d\") " pod="ai-tenant-e2e-shared-b-046c73/auth-test-model-046c73-kserve-858d9d9d87-46dtw" Jul 20 21:46:42.983117 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:42.983070 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ef46531-e365-460e-addf-02999d90790d-tmp-dir\") pod \"auth-test-model-046c73-kserve-858d9d9d87-46dtw\" (UID: \"4ef46531-e365-460e-addf-02999d90790d\") " pod="ai-tenant-e2e-shared-b-046c73/auth-test-model-046c73-kserve-858d9d9d87-46dtw" Jul 20 21:46:42.983117 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:42.983100 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ef46531-e365-460e-addf-02999d90790d-tls-certs\") pod \"auth-test-model-046c73-kserve-858d9d9d87-46dtw\" (UID: \"4ef46531-e365-460e-addf-02999d90790d\") " pod="ai-tenant-e2e-shared-b-046c73/auth-test-model-046c73-kserve-858d9d9d87-46dtw" Jul 20 21:46:42.983278 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:42.983128 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ef46531-e365-460e-addf-02999d90790d-home\") pod \"auth-test-model-046c73-kserve-858d9d9d87-46dtw\" (UID: \"4ef46531-e365-460e-addf-02999d90790d\") " pod="ai-tenant-e2e-shared-b-046c73/auth-test-model-046c73-kserve-858d9d9d87-46dtw" Jul 20 21:46:42.983462 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:42.983432 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ef46531-e365-460e-addf-02999d90790d-model-cache\") pod \"auth-test-model-046c73-kserve-858d9d9d87-46dtw\" (UID: \"4ef46531-e365-460e-addf-02999d90790d\") " pod="ai-tenant-e2e-shared-b-046c73/auth-test-model-046c73-kserve-858d9d9d87-46dtw" Jul 20 21:46:42.983574 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:42.983482 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ef46531-e365-460e-addf-02999d90790d-home\") pod \"auth-test-model-046c73-kserve-858d9d9d87-46dtw\" (UID: \"4ef46531-e365-460e-addf-02999d90790d\") " pod="ai-tenant-e2e-shared-b-046c73/auth-test-model-046c73-kserve-858d9d9d87-46dtw" Jul 20 21:46:42.983574 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:42.983518 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ef46531-e365-460e-addf-02999d90790d-tmp-dir\") pod \"auth-test-model-046c73-kserve-858d9d9d87-46dtw\" (UID: \"4ef46531-e365-460e-addf-02999d90790d\") " pod="ai-tenant-e2e-shared-b-046c73/auth-test-model-046c73-kserve-858d9d9d87-46dtw" Jul 20 21:46:42.985244 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:42.985225 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ef46531-e365-460e-addf-02999d90790d-dshm\") pod \"auth-test-model-046c73-kserve-858d9d9d87-46dtw\" (UID: \"4ef46531-e365-460e-addf-02999d90790d\") " pod="ai-tenant-e2e-shared-b-046c73/auth-test-model-046c73-kserve-858d9d9d87-46dtw" Jul 20 21:46:42.985504 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:42.985485 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ef46531-e365-460e-addf-02999d90790d-tls-certs\") pod \"auth-test-model-046c73-kserve-858d9d9d87-46dtw\" (UID: \"4ef46531-e365-460e-addf-02999d90790d\") " pod="ai-tenant-e2e-shared-b-046c73/auth-test-model-046c73-kserve-858d9d9d87-46dtw" Jul 20 21:46:42.993187 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:42.993165 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbcn8\" (UniqueName: \"kubernetes.io/projected/4ef46531-e365-460e-addf-02999d90790d-kube-api-access-lbcn8\") pod \"auth-test-model-046c73-kserve-858d9d9d87-46dtw\" (UID: \"4ef46531-e365-460e-addf-02999d90790d\") " pod="ai-tenant-e2e-shared-b-046c73/auth-test-model-046c73-kserve-858d9d9d87-46dtw" Jul 20 21:46:43.103661 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:43.103572 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-046c73/auth-test-model-046c73-kserve-858d9d9d87-46dtw" Jul 20 21:46:43.242216 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:43.242127 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-046c73/auth-test-model-046c73-kserve-858d9d9d87-46dtw"] Jul 20 21:46:43.244471 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:46:43.244443 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ef46531_e365_460e_addf_02999d90790d.slice/crio-e01be67a537c874507583d9c1f903f3b35af3f8ce3cafc5c2f100eff0a44853a WatchSource:0}: Error finding container e01be67a537c874507583d9c1f903f3b35af3f8ce3cafc5c2f100eff0a44853a: Status 404 returned error can't find the container with id e01be67a537c874507583d9c1f903f3b35af3f8ce3cafc5c2f100eff0a44853a Jul 20 21:46:43.417904 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:43.417798 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-046c73/auth-test-model-046c73-kserve-858d9d9d87-46dtw" event={"ID":"4ef46531-e365-460e-addf-02999d90790d","Type":"ContainerStarted","Data":"825b2dc110f82983d9c814c3f2876b5e41dedf09bbf450fa3cbff7a4e6bfe8aa"} Jul 20 21:46:43.417904 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:43.417844 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-046c73/auth-test-model-046c73-kserve-858d9d9d87-46dtw" event={"ID":"4ef46531-e365-460e-addf-02999d90790d","Type":"ContainerStarted","Data":"e01be67a537c874507583d9c1f903f3b35af3f8ce3cafc5c2f100eff0a44853a"} Jul 20 21:46:43.443696 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:43.443645 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-046c73/auth-test-model-046c73-kserve-858d9d9d87-46dtw" podStartSLOduration=1.443621064 podStartE2EDuration="1.443621064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 21:46:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:46:43.441511572 +0000 UTC m=+2642.767214514" watchObservedRunningTime="2026-07-20 21:46:43.443621064 +0000 UTC m=+2642.769324006" Jul 20 21:46:53.104541 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:53.104505 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-046c73/auth-test-model-046c73-kserve-858d9d9d87-46dtw" Jul 20 21:46:53.104943 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:53.104559 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-046c73/auth-test-model-046c73-kserve-858d9d9d87-46dtw" Jul 20 21:46:53.117215 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:53.117190 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-046c73/auth-test-model-046c73-kserve-858d9d9d87-46dtw" Jul 20 21:46:53.482174 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:46:53.482083 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-046c73/auth-test-model-046c73-kserve-858d9d9d87-46dtw" Jul 20 21:48:03.455597 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:48:03.455514 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:48:03.556389 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:48:03.556356 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:48:51.988661 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:48:51.988611 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:48:52.974283 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:48:52.974248 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-046c73/auth-test-model-046c73-kserve-858d9d9d87-46dtw"] Jul 20 21:48:52.994344 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:48:52.994317 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:49:07.994848 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:07.994732 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-046c73/auth-test-model-046c73-kserve-858d9d9d87-46dtw" podUID="4ef46531-e365-460e-addf-02999d90790d" containerName="main" containerID="cri-o://825b2dc110f82983d9c814c3f2876b5e41dedf09bbf450fa3cbff7a4e6bfe8aa" gracePeriod=45 Jul 20 21:49:08.189185 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:08.189147 2562 generic.go:356] "Generic (PLEG): container finished" podID="4ef46531-e365-460e-addf-02999d90790d" containerID="825b2dc110f82983d9c814c3f2876b5e41dedf09bbf450fa3cbff7a4e6bfe8aa" exitCode=0 Jul 20 21:49:08.189185 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:08.189171 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-046c73/auth-test-model-046c73-kserve-858d9d9d87-46dtw" event={"ID":"4ef46531-e365-460e-addf-02999d90790d","Type":"ContainerDied","Data":"825b2dc110f82983d9c814c3f2876b5e41dedf09bbf450fa3cbff7a4e6bfe8aa"} Jul 20 21:49:08.249087 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:08.249029 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-046c73/auth-test-model-046c73-kserve-858d9d9d87-46dtw" Jul 20 21:49:08.370881 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:08.370844 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ef46531-e365-460e-addf-02999d90790d-home\") pod \"4ef46531-e365-460e-addf-02999d90790d\" (UID: \"4ef46531-e365-460e-addf-02999d90790d\") " Jul 20 21:49:08.370881 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:08.370886 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ef46531-e365-460e-addf-02999d90790d-dshm\") pod \"4ef46531-e365-460e-addf-02999d90790d\" (UID: \"4ef46531-e365-460e-addf-02999d90790d\") " Jul 20 21:49:08.371127 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:08.370914 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbcn8\" (UniqueName: \"kubernetes.io/projected/4ef46531-e365-460e-addf-02999d90790d-kube-api-access-lbcn8\") pod \"4ef46531-e365-460e-addf-02999d90790d\" (UID: \"4ef46531-e365-460e-addf-02999d90790d\") " Jul 20 21:49:08.371127 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:08.370963 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ef46531-e365-460e-addf-02999d90790d-tmp-dir\") pod \"4ef46531-e365-460e-addf-02999d90790d\" (UID: \"4ef46531-e365-460e-addf-02999d90790d\") " Jul 20 21:49:08.371127 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:08.370989 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ef46531-e365-460e-addf-02999d90790d-model-cache\") pod \"4ef46531-e365-460e-addf-02999d90790d\" (UID: \"4ef46531-e365-460e-addf-02999d90790d\") " Jul 20 21:49:08.371127 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:08.371006 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ef46531-e365-460e-addf-02999d90790d-tls-certs\") pod \"4ef46531-e365-460e-addf-02999d90790d\" (UID: \"4ef46531-e365-460e-addf-02999d90790d\") " Jul 20 21:49:08.371376 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:08.371188 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ef46531-e365-460e-addf-02999d90790d-home" (OuterVolumeSpecName: "home") pod "4ef46531-e365-460e-addf-02999d90790d" (UID: "4ef46531-e365-460e-addf-02999d90790d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 21:49:08.371376 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:08.371309 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ef46531-e365-460e-addf-02999d90790d-model-cache" (OuterVolumeSpecName: "model-cache") pod "4ef46531-e365-460e-addf-02999d90790d" (UID: "4ef46531-e365-460e-addf-02999d90790d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 21:49:08.371376 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:08.371311 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ef46531-e365-460e-addf-02999d90790d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4ef46531-e365-460e-addf-02999d90790d" (UID: "4ef46531-e365-460e-addf-02999d90790d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 21:49:08.371533 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:08.371379 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ef46531-e365-460e-addf-02999d90790d-home\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:49:08.371533 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:08.371395 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ef46531-e365-460e-addf-02999d90790d-tmp-dir\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:49:08.371533 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:08.371403 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ef46531-e365-460e-addf-02999d90790d-model-cache\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:49:08.373031 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:08.373012 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ef46531-e365-460e-addf-02999d90790d-dshm" (OuterVolumeSpecName: "dshm") pod "4ef46531-e365-460e-addf-02999d90790d" (UID: "4ef46531-e365-460e-addf-02999d90790d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 21:49:08.373193 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:08.373173 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ef46531-e365-460e-addf-02999d90790d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4ef46531-e365-460e-addf-02999d90790d" (UID: "4ef46531-e365-460e-addf-02999d90790d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 21:49:08.373257 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:08.373238 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ef46531-e365-460e-addf-02999d90790d-kube-api-access-lbcn8" (OuterVolumeSpecName: "kube-api-access-lbcn8") pod "4ef46531-e365-460e-addf-02999d90790d" (UID: "4ef46531-e365-460e-addf-02999d90790d"). InnerVolumeSpecName "kube-api-access-lbcn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 21:49:08.472057 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:08.472020 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ef46531-e365-460e-addf-02999d90790d-dshm\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:49:08.472057 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:08.472051 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbcn8\" (UniqueName: \"kubernetes.io/projected/4ef46531-e365-460e-addf-02999d90790d-kube-api-access-lbcn8\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:49:08.472057 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:08.472063 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ef46531-e365-460e-addf-02999d90790d-tls-certs\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:49:09.195375 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:09.195350 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-046c73/auth-test-model-046c73-kserve-858d9d9d87-46dtw" Jul 20 21:49:09.195375 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:09.195360 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-046c73/auth-test-model-046c73-kserve-858d9d9d87-46dtw" event={"ID":"4ef46531-e365-460e-addf-02999d90790d","Type":"ContainerDied","Data":"e01be67a537c874507583d9c1f903f3b35af3f8ce3cafc5c2f100eff0a44853a"} Jul 20 21:49:09.197922 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:09.195403 2562 scope.go:117] "RemoveContainer" containerID="825b2dc110f82983d9c814c3f2876b5e41dedf09bbf450fa3cbff7a4e6bfe8aa" Jul 20 21:49:09.220070 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:09.220047 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-046c73/auth-test-model-046c73-kserve-858d9d9d87-46dtw"] Jul 20 21:49:09.226182 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:09.226161 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-046c73/auth-test-model-046c73-kserve-858d9d9d87-46dtw"] Jul 20 21:49:09.270550 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:09.270526 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ef46531-e365-460e-addf-02999d90790d" path="/var/lib/kubelet/pods/4ef46531-e365-460e-addf-02999d90790d/volumes" Jul 20 21:49:12.532829 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:12.532796 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:49:12.630923 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:12.630877 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:49:19.335044 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:19.335009 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:49:26.314555 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:26.314518 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:49:46.112293 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:46.112261 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-046c73/sub-test-model-046c73-kserve-6c5f5bb477-p4gzt"] Jul 20 21:49:46.112683 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:46.112669 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ef46531-e365-460e-addf-02999d90790d" containerName="main" Jul 20 21:49:46.112726 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:46.112684 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ef46531-e365-460e-addf-02999d90790d" containerName="main" Jul 20 21:49:46.112775 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:46.112766 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ef46531-e365-460e-addf-02999d90790d" containerName="main" Jul 20 21:49:46.117297 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:46.117279 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-046c73/sub-test-model-046c73-kserve-6c5f5bb477-p4gzt" Jul 20 21:49:46.121402 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:46.121380 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-046c73\"/\"kube-root-ca.crt\"" Jul 20 21:49:46.121549 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:46.121382 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-046c73\"/\"default-dockercfg-rwqs9\"" Jul 20 21:49:46.121549 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:46.121419 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-046c73\"/\"sub-test-model-046c73-kserve-self-signed-certs\"" Jul 20 21:49:46.121549 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:46.121446 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-046c73\"/\"openshift-service-ca.crt\"" Jul 20 21:49:46.130244 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:46.130221 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-046c73/sub-test-model-046c73-kserve-6c5f5bb477-p4gzt"] Jul 20 21:49:46.152599 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:46.152571 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:49:46.233658 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:46.233628 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/292ab8f3-da17-4718-831c-36d60614d987-dshm\") pod \"sub-test-model-046c73-kserve-6c5f5bb477-p4gzt\" (UID: \"292ab8f3-da17-4718-831c-36d60614d987\") " pod="ai-tenant-e2e-shared-b-046c73/sub-test-model-046c73-kserve-6c5f5bb477-p4gzt" Jul 20 21:49:46.233816 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:46.233667 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/292ab8f3-da17-4718-831c-36d60614d987-model-cache\") pod \"sub-test-model-046c73-kserve-6c5f5bb477-p4gzt\" (UID: \"292ab8f3-da17-4718-831c-36d60614d987\") " pod="ai-tenant-e2e-shared-b-046c73/sub-test-model-046c73-kserve-6c5f5bb477-p4gzt" Jul 20 21:49:46.233816 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:46.233688 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/292ab8f3-da17-4718-831c-36d60614d987-tls-certs\") pod \"sub-test-model-046c73-kserve-6c5f5bb477-p4gzt\" (UID: \"292ab8f3-da17-4718-831c-36d60614d987\") " pod="ai-tenant-e2e-shared-b-046c73/sub-test-model-046c73-kserve-6c5f5bb477-p4gzt" Jul 20 21:49:46.233816 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:46.233770 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dppfd\" (UniqueName: \"kubernetes.io/projected/292ab8f3-da17-4718-831c-36d60614d987-kube-api-access-dppfd\") pod \"sub-test-model-046c73-kserve-6c5f5bb477-p4gzt\" (UID: \"292ab8f3-da17-4718-831c-36d60614d987\") " pod="ai-tenant-e2e-shared-b-046c73/sub-test-model-046c73-kserve-6c5f5bb477-p4gzt" Jul 20 21:49:46.233816 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:46.233806 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/292ab8f3-da17-4718-831c-36d60614d987-tmp-dir\") pod \"sub-test-model-046c73-kserve-6c5f5bb477-p4gzt\" (UID: \"292ab8f3-da17-4718-831c-36d60614d987\") " pod="ai-tenant-e2e-shared-b-046c73/sub-test-model-046c73-kserve-6c5f5bb477-p4gzt" Jul 20 21:49:46.233971 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:46.233835 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/292ab8f3-da17-4718-831c-36d60614d987-home\") pod \"sub-test-model-046c73-kserve-6c5f5bb477-p4gzt\" (UID: \"292ab8f3-da17-4718-831c-36d60614d987\") " pod="ai-tenant-e2e-shared-b-046c73/sub-test-model-046c73-kserve-6c5f5bb477-p4gzt" Jul 20 21:49:46.335071 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:46.335033 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/292ab8f3-da17-4718-831c-36d60614d987-model-cache\") pod \"sub-test-model-046c73-kserve-6c5f5bb477-p4gzt\" (UID: \"292ab8f3-da17-4718-831c-36d60614d987\") " pod="ai-tenant-e2e-shared-b-046c73/sub-test-model-046c73-kserve-6c5f5bb477-p4gzt" Jul 20 21:49:46.335071 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:46.335074 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/292ab8f3-da17-4718-831c-36d60614d987-tls-certs\") pod \"sub-test-model-046c73-kserve-6c5f5bb477-p4gzt\" (UID: \"292ab8f3-da17-4718-831c-36d60614d987\") " pod="ai-tenant-e2e-shared-b-046c73/sub-test-model-046c73-kserve-6c5f5bb477-p4gzt" Jul 20 21:49:46.335319 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:46.335107 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dppfd\" (UniqueName: \"kubernetes.io/projected/292ab8f3-da17-4718-831c-36d60614d987-kube-api-access-dppfd\") pod \"sub-test-model-046c73-kserve-6c5f5bb477-p4gzt\" (UID: \"292ab8f3-da17-4718-831c-36d60614d987\") " pod="ai-tenant-e2e-shared-b-046c73/sub-test-model-046c73-kserve-6c5f5bb477-p4gzt" Jul 20 21:49:46.335319 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:46.335128 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/292ab8f3-da17-4718-831c-36d60614d987-tmp-dir\") pod \"sub-test-model-046c73-kserve-6c5f5bb477-p4gzt\" (UID: \"292ab8f3-da17-4718-831c-36d60614d987\") " pod="ai-tenant-e2e-shared-b-046c73/sub-test-model-046c73-kserve-6c5f5bb477-p4gzt" Jul 20 21:49:46.335319 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:46.335174 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/292ab8f3-da17-4718-831c-36d60614d987-home\") pod \"sub-test-model-046c73-kserve-6c5f5bb477-p4gzt\" (UID: \"292ab8f3-da17-4718-831c-36d60614d987\") " pod="ai-tenant-e2e-shared-b-046c73/sub-test-model-046c73-kserve-6c5f5bb477-p4gzt" Jul 20 21:49:46.335319 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:46.335234 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/292ab8f3-da17-4718-831c-36d60614d987-dshm\") pod \"sub-test-model-046c73-kserve-6c5f5bb477-p4gzt\" (UID: \"292ab8f3-da17-4718-831c-36d60614d987\") " pod="ai-tenant-e2e-shared-b-046c73/sub-test-model-046c73-kserve-6c5f5bb477-p4gzt" Jul 20 21:49:46.335553 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:46.335529 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/292ab8f3-da17-4718-831c-36d60614d987-model-cache\") pod \"sub-test-model-046c73-kserve-6c5f5bb477-p4gzt\" (UID: \"292ab8f3-da17-4718-831c-36d60614d987\") " pod="ai-tenant-e2e-shared-b-046c73/sub-test-model-046c73-kserve-6c5f5bb477-p4gzt" Jul 20 21:49:46.335632 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:46.335562 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/292ab8f3-da17-4718-831c-36d60614d987-tmp-dir\") pod \"sub-test-model-046c73-kserve-6c5f5bb477-p4gzt\" (UID: \"292ab8f3-da17-4718-831c-36d60614d987\") " pod="ai-tenant-e2e-shared-b-046c73/sub-test-model-046c73-kserve-6c5f5bb477-p4gzt" Jul 20 21:49:46.335690 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:46.335658 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/292ab8f3-da17-4718-831c-36d60614d987-home\") pod \"sub-test-model-046c73-kserve-6c5f5bb477-p4gzt\" (UID: \"292ab8f3-da17-4718-831c-36d60614d987\") " pod="ai-tenant-e2e-shared-b-046c73/sub-test-model-046c73-kserve-6c5f5bb477-p4gzt" Jul 20 21:49:46.337463 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:46.337445 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/292ab8f3-da17-4718-831c-36d60614d987-dshm\") pod \"sub-test-model-046c73-kserve-6c5f5bb477-p4gzt\" (UID: \"292ab8f3-da17-4718-831c-36d60614d987\") " pod="ai-tenant-e2e-shared-b-046c73/sub-test-model-046c73-kserve-6c5f5bb477-p4gzt" Jul 20 21:49:46.337646 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:46.337627 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/292ab8f3-da17-4718-831c-36d60614d987-tls-certs\") pod \"sub-test-model-046c73-kserve-6c5f5bb477-p4gzt\" (UID: \"292ab8f3-da17-4718-831c-36d60614d987\") " pod="ai-tenant-e2e-shared-b-046c73/sub-test-model-046c73-kserve-6c5f5bb477-p4gzt" Jul 20 21:49:46.345300 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:46.345276 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dppfd\" (UniqueName: \"kubernetes.io/projected/292ab8f3-da17-4718-831c-36d60614d987-kube-api-access-dppfd\") pod \"sub-test-model-046c73-kserve-6c5f5bb477-p4gzt\" (UID: \"292ab8f3-da17-4718-831c-36d60614d987\") " pod="ai-tenant-e2e-shared-b-046c73/sub-test-model-046c73-kserve-6c5f5bb477-p4gzt" Jul 20 21:49:46.427122 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:46.427037 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-046c73/sub-test-model-046c73-kserve-6c5f5bb477-p4gzt" Jul 20 21:49:46.577767 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:46.577742 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-046c73/sub-test-model-046c73-kserve-6c5f5bb477-p4gzt"] Jul 20 21:49:46.580419 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:49:46.580393 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod292ab8f3_da17_4718_831c_36d60614d987.slice/crio-378f5d8e1d56be07d9f936e557f325f058653d519602b1ce61d511874e8c9c4d WatchSource:0}: Error finding container 378f5d8e1d56be07d9f936e557f325f058653d519602b1ce61d511874e8c9c4d: Status 404 returned error can't find the container with id 378f5d8e1d56be07d9f936e557f325f058653d519602b1ce61d511874e8c9c4d Jul 20 21:49:47.388443 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:47.388408 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-046c73/sub-test-model-046c73-kserve-6c5f5bb477-p4gzt" event={"ID":"292ab8f3-da17-4718-831c-36d60614d987","Type":"ContainerStarted","Data":"fe3781ff078ff0f7fc25d54c2ac5ed6f476b08f22ac5171d90745b5c1fb501e3"} Jul 20 21:49:47.388871 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:47.388453 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-046c73/sub-test-model-046c73-kserve-6c5f5bb477-p4gzt" event={"ID":"292ab8f3-da17-4718-831c-36d60614d987","Type":"ContainerStarted","Data":"378f5d8e1d56be07d9f936e557f325f058653d519602b1ce61d511874e8c9c4d"} Jul 20 21:49:47.410455 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:47.410411 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-046c73/sub-test-model-046c73-kserve-6c5f5bb477-p4gzt" podStartSLOduration=1.410397212 podStartE2EDuration="1.410397212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 21:49:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:49:47.409633308 +0000 UTC m=+2826.735336251" watchObservedRunningTime="2026-07-20 21:49:47.410397212 +0000 UTC m=+2826.736100219" Jul 20 21:49:56.427679 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:56.427633 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-046c73/sub-test-model-046c73-kserve-6c5f5bb477-p4gzt" Jul 20 21:49:56.428198 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:56.427704 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-046c73/sub-test-model-046c73-kserve-6c5f5bb477-p4gzt" Jul 20 21:49:56.440350 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:56.440324 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-046c73/sub-test-model-046c73-kserve-6c5f5bb477-p4gzt" Jul 20 21:49:56.451435 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:56.451412 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-046c73/sub-test-model-046c73-kserve-6c5f5bb477-p4gzt" Jul 20 21:49:56.553946 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:56.553912 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:49:56.654269 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:49:56.654235 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:50:18.718810 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:50:18.718773 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:50:34.550017 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:50:34.549906 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:50:35.373902 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:50:35.373870 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-046c73/sub-test-model-046c73-kserve-6c5f5bb477-p4gzt"] Jul 20 21:50:35.416327 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:50:35.416296 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:50:39.276075 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:50:39.276037 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:50:41.447371 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:50:41.447339 2562 scope.go:117] "RemoveContainer" containerID="22a5af851e42405a8ed0508fd7e9f8644ad957703f01572a5a5b4b421183c99f" Jul 20 21:50:46.954602 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:50:46.954566 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:50:47.057610 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:50:47.057581 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:50:50.393720 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:50:50.393685 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-046c73/sub-test-model-046c73-kserve-6c5f5bb477-p4gzt" podUID="292ab8f3-da17-4718-831c-36d60614d987" containerName="main" containerID="cri-o://fe3781ff078ff0f7fc25d54c2ac5ed6f476b08f22ac5171d90745b5c1fb501e3" gracePeriod=45 Jul 20 21:50:50.640268 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:50:50.640244 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-046c73/sub-test-model-046c73-kserve-6c5f5bb477-p4gzt" Jul 20 21:50:50.708974 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:50:50.708882 2562 generic.go:356] "Generic (PLEG): container finished" podID="292ab8f3-da17-4718-831c-36d60614d987" containerID="fe3781ff078ff0f7fc25d54c2ac5ed6f476b08f22ac5171d90745b5c1fb501e3" exitCode=0 Jul 20 21:50:50.708974 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:50:50.708929 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-046c73/sub-test-model-046c73-kserve-6c5f5bb477-p4gzt" Jul 20 21:50:50.708974 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:50:50.708950 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-046c73/sub-test-model-046c73-kserve-6c5f5bb477-p4gzt" event={"ID":"292ab8f3-da17-4718-831c-36d60614d987","Type":"ContainerDied","Data":"fe3781ff078ff0f7fc25d54c2ac5ed6f476b08f22ac5171d90745b5c1fb501e3"} Jul 20 21:50:50.709262 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:50:50.708983 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-046c73/sub-test-model-046c73-kserve-6c5f5bb477-p4gzt" event={"ID":"292ab8f3-da17-4718-831c-36d60614d987","Type":"ContainerDied","Data":"378f5d8e1d56be07d9f936e557f325f058653d519602b1ce61d511874e8c9c4d"} Jul 20 21:50:50.709262 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:50:50.709001 2562 scope.go:117] "RemoveContainer" containerID="fe3781ff078ff0f7fc25d54c2ac5ed6f476b08f22ac5171d90745b5c1fb501e3" Jul 20 21:50:50.717605 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:50:50.717588 2562 scope.go:117] "RemoveContainer" containerID="fe3781ff078ff0f7fc25d54c2ac5ed6f476b08f22ac5171d90745b5c1fb501e3" Jul 20 21:50:50.717873 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:50:50.717852 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe3781ff078ff0f7fc25d54c2ac5ed6f476b08f22ac5171d90745b5c1fb501e3\": container with ID starting with fe3781ff078ff0f7fc25d54c2ac5ed6f476b08f22ac5171d90745b5c1fb501e3 not found: ID does not exist" containerID="fe3781ff078ff0f7fc25d54c2ac5ed6f476b08f22ac5171d90745b5c1fb501e3" Jul 20 21:50:50.717982 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:50:50.717879 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe3781ff078ff0f7fc25d54c2ac5ed6f476b08f22ac5171d90745b5c1fb501e3"} err="failed to get container status \"fe3781ff078ff0f7fc25d54c2ac5ed6f476b08f22ac5171d90745b5c1fb501e3\": rpc error: code = NotFound desc = could not find container \"fe3781ff078ff0f7fc25d54c2ac5ed6f476b08f22ac5171d90745b5c1fb501e3\": container with ID starting with fe3781ff078ff0f7fc25d54c2ac5ed6f476b08f22ac5171d90745b5c1fb501e3 not found: ID does not exist" Jul 20 21:50:50.741253 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:50:50.741224 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/292ab8f3-da17-4718-831c-36d60614d987-tmp-dir\") pod \"292ab8f3-da17-4718-831c-36d60614d987\" (UID: \"292ab8f3-da17-4718-831c-36d60614d987\") " Jul 20 21:50:50.741421 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:50:50.741278 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/292ab8f3-da17-4718-831c-36d60614d987-tls-certs\") pod \"292ab8f3-da17-4718-831c-36d60614d987\" (UID: \"292ab8f3-da17-4718-831c-36d60614d987\") " Jul 20 21:50:50.741421 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:50:50.741349 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/292ab8f3-da17-4718-831c-36d60614d987-model-cache\") pod \"292ab8f3-da17-4718-831c-36d60614d987\" (UID: \"292ab8f3-da17-4718-831c-36d60614d987\") " Jul 20 21:50:50.741421 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:50:50.741367 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/292ab8f3-da17-4718-831c-36d60614d987-home\") pod \"292ab8f3-da17-4718-831c-36d60614d987\" (UID: \"292ab8f3-da17-4718-831c-36d60614d987\") " Jul 20 21:50:50.741421 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:50:50.741404 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dppfd\" (UniqueName: \"kubernetes.io/projected/292ab8f3-da17-4718-831c-36d60614d987-kube-api-access-dppfd\") pod \"292ab8f3-da17-4718-831c-36d60614d987\" (UID: \"292ab8f3-da17-4718-831c-36d60614d987\") " Jul 20 21:50:50.741644 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:50:50.741443 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/292ab8f3-da17-4718-831c-36d60614d987-dshm\") pod \"292ab8f3-da17-4718-831c-36d60614d987\" (UID: \"292ab8f3-da17-4718-831c-36d60614d987\") " Jul 20 21:50:50.741644 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:50:50.741502 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/292ab8f3-da17-4718-831c-36d60614d987-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "292ab8f3-da17-4718-831c-36d60614d987" (UID: "292ab8f3-da17-4718-831c-36d60614d987"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 21:50:50.741644 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:50:50.741632 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/292ab8f3-da17-4718-831c-36d60614d987-home" (OuterVolumeSpecName: "home") pod "292ab8f3-da17-4718-831c-36d60614d987" (UID: "292ab8f3-da17-4718-831c-36d60614d987"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 21:50:50.741786 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:50:50.741637 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/292ab8f3-da17-4718-831c-36d60614d987-model-cache" (OuterVolumeSpecName: "model-cache") pod "292ab8f3-da17-4718-831c-36d60614d987" (UID: "292ab8f3-da17-4718-831c-36d60614d987"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 21:50:50.741828 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:50:50.741783 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/292ab8f3-da17-4718-831c-36d60614d987-model-cache\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:50:50.741828 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:50:50.741802 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/292ab8f3-da17-4718-831c-36d60614d987-home\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:50:50.741905 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:50:50.741840 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/292ab8f3-da17-4718-831c-36d60614d987-tmp-dir\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:50:50.743579 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:50:50.743545 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/292ab8f3-da17-4718-831c-36d60614d987-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "292ab8f3-da17-4718-831c-36d60614d987" (UID: "292ab8f3-da17-4718-831c-36d60614d987"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 21:50:50.743579 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:50:50.743561 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/292ab8f3-da17-4718-831c-36d60614d987-dshm" (OuterVolumeSpecName: "dshm") pod "292ab8f3-da17-4718-831c-36d60614d987" (UID: "292ab8f3-da17-4718-831c-36d60614d987"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 21:50:50.743756 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:50:50.743579 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/292ab8f3-da17-4718-831c-36d60614d987-kube-api-access-dppfd" (OuterVolumeSpecName: "kube-api-access-dppfd") pod "292ab8f3-da17-4718-831c-36d60614d987" (UID: "292ab8f3-da17-4718-831c-36d60614d987"). InnerVolumeSpecName "kube-api-access-dppfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 21:50:50.842910 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:50:50.842875 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/292ab8f3-da17-4718-831c-36d60614d987-dshm\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:50:50.842910 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:50:50.842904 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/292ab8f3-da17-4718-831c-36d60614d987-tls-certs\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:50:50.842910 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:50:50.842915 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dppfd\" (UniqueName: \"kubernetes.io/projected/292ab8f3-da17-4718-831c-36d60614d987-kube-api-access-dppfd\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:50:51.032791 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:50:51.032755 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-046c73/sub-test-model-046c73-kserve-6c5f5bb477-p4gzt"] Jul 20 21:50:51.037531 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:50:51.037499 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-046c73/sub-test-model-046c73-kserve-6c5f5bb477-p4gzt"] Jul 20 21:50:51.270585 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:50:51.270495 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="292ab8f3-da17-4718-831c-36d60614d987" path="/var/lib/kubelet/pods/292ab8f3-da17-4718-831c-36d60614d987/volumes" Jul 20 21:50:56.726896 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:50:56.726865 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:51:26.978786 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:26.978752 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:51:27.080739 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:27.080698 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:51:33.798716 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:33.798657 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:51:33.898343 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:33.898312 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:51:41.089425 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:41.089389 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:51:41.467667 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:41.467586 2562 scope.go:117] "RemoveContainer" containerID="290f2d68dcf3608a730cd8b2b0ff8797fc67474c93a383bffd38e65160d0f937" Jul 20 21:51:41.475772 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:41.475743 2562 scope.go:117] "RemoveContainer" containerID="6cbe4da4252b5c28cef3a9f432f8f4951f2311ab346ff6491571e721dd74ef69" Jul 20 21:51:41.486268 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:41.486251 2562 scope.go:117] "RemoveContainer" containerID="d1aebf6249bbe4f8c0a2dc7dae6a56ebff00533cb9ece3d640413545c2f7efbe" Jul 20 21:51:41.495761 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:41.495743 2562 scope.go:117] "RemoveContainer" containerID="51c25a5840fac7f57894fe00ce5d47e4ad579593ec2146f9d9b25738f8537386" Jul 20 21:51:41.997114 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:41.997082 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:51:42.939170 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:42.939120 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7634-140077526541360"] Jul 20 21:51:42.939589 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:42.939573 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="292ab8f3-da17-4718-831c-36d60614d987" containerName="main" Jul 20 21:51:42.939649 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:42.939591 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="292ab8f3-da17-4718-831c-36d60614d987" containerName="main" Jul 20 21:51:42.939689 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:42.939674 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="292ab8f3-da17-4718-831c-36d60614d987" containerName="main" Jul 20 21:51:42.944723 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:42.944700 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7634-140077526541360" Jul 20 21:51:42.947938 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:42.947916 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-lhmnw\"" Jul 20 21:51:42.954509 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:42.954489 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7634-140077526541360"] Jul 20 21:51:43.027734 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:43.027700 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d88vc\" (UniqueName: \"kubernetes.io/projected/01c1f8e6-3559-4ef8-adbb-94617120f3bc-kube-api-access-d88vc\") pod \"test-curl-7634-140077526541360\" (UID: \"01c1f8e6-3559-4ef8-adbb-94617120f3bc\") " pod="opendatahub/test-curl-7634-140077526541360" Jul 20 21:51:43.128738 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:43.128701 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d88vc\" (UniqueName: \"kubernetes.io/projected/01c1f8e6-3559-4ef8-adbb-94617120f3bc-kube-api-access-d88vc\") pod \"test-curl-7634-140077526541360\" (UID: \"01c1f8e6-3559-4ef8-adbb-94617120f3bc\") " pod="opendatahub/test-curl-7634-140077526541360" Jul 20 21:51:43.141020 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:43.140992 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d88vc\" (UniqueName: \"kubernetes.io/projected/01c1f8e6-3559-4ef8-adbb-94617120f3bc-kube-api-access-d88vc\") pod \"test-curl-7634-140077526541360\" (UID: \"01c1f8e6-3559-4ef8-adbb-94617120f3bc\") " pod="opendatahub/test-curl-7634-140077526541360" Jul 20 21:51:43.254739 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:43.254657 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7634-140077526541360" Jul 20 21:51:43.603909 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:43.603884 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7634-140077526541360"] Jul 20 21:51:43.605801 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:51:43.605773 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod01c1f8e6_3559_4ef8_adbb_94617120f3bc.slice/crio-d8fa6fd6d3a8a3d8919221ea6f46bb6e1a3862827fbe4c6da6008219150a9b28 WatchSource:0}: Error finding container d8fa6fd6d3a8a3d8919221ea6f46bb6e1a3862827fbe4c6da6008219150a9b28: Status 404 returned error can't find the container with id d8fa6fd6d3a8a3d8919221ea6f46bb6e1a3862827fbe4c6da6008219150a9b28 Jul 20 21:51:43.607117 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:43.607102 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 21:51:43.987813 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:43.987724 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7634-140077526541360" event={"ID":"01c1f8e6-3559-4ef8-adbb-94617120f3bc","Type":"ContainerStarted","Data":"d8fa6fd6d3a8a3d8919221ea6f46bb6e1a3862827fbe4c6da6008219150a9b28"} Jul 20 21:51:44.993691 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:44.993657 2562 generic.go:356] "Generic (PLEG): container finished" podID="01c1f8e6-3559-4ef8-adbb-94617120f3bc" containerID="412bfe0b9cabf9aa6f20226f21d246f42dc7d584bf9ea909a8e77be260af7493" exitCode=0 Jul 20 21:51:44.994117 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:44.993754 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7634-140077526541360" event={"ID":"01c1f8e6-3559-4ef8-adbb-94617120f3bc","Type":"ContainerDied","Data":"412bfe0b9cabf9aa6f20226f21d246f42dc7d584bf9ea909a8e77be260af7493"} Jul 20 21:51:46.133781 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:46.133759 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7634-140077526541360" Jul 20 21:51:46.154194 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:46.154154 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7634-140077526541360_01c1f8e6-3559-4ef8-adbb-94617120f3bc/test-curl-7634-140077526541360/0.log" Jul 20 21:51:46.157439 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:46.157417 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d88vc\" (UniqueName: \"kubernetes.io/projected/01c1f8e6-3559-4ef8-adbb-94617120f3bc-kube-api-access-d88vc\") pod \"01c1f8e6-3559-4ef8-adbb-94617120f3bc\" (UID: \"01c1f8e6-3559-4ef8-adbb-94617120f3bc\") " Jul 20 21:51:46.159493 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:46.159468 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01c1f8e6-3559-4ef8-adbb-94617120f3bc-kube-api-access-d88vc" (OuterVolumeSpecName: "kube-api-access-d88vc") pod "01c1f8e6-3559-4ef8-adbb-94617120f3bc" (UID: "01c1f8e6-3559-4ef8-adbb-94617120f3bc"). InnerVolumeSpecName "kube-api-access-d88vc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 21:51:46.183550 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:46.183527 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7634-140077526541360"] Jul 20 21:51:46.188470 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:46.188446 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7634-140077526541360"] Jul 20 21:51:46.258555 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:46.258530 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d88vc\" (UniqueName: \"kubernetes.io/projected/01c1f8e6-3559-4ef8-adbb-94617120f3bc-kube-api-access-d88vc\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:51:46.308508 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:46.308476 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7634-140077526542768"] Jul 20 21:51:46.308896 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:46.308880 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01c1f8e6-3559-4ef8-adbb-94617120f3bc" containerName="test-curl-7634-140077526541360" Jul 20 21:51:46.308943 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:46.308897 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="01c1f8e6-3559-4ef8-adbb-94617120f3bc" containerName="test-curl-7634-140077526541360" Jul 20 21:51:46.308989 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:46.308979 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="01c1f8e6-3559-4ef8-adbb-94617120f3bc" containerName="test-curl-7634-140077526541360" Jul 20 21:51:46.311934 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:46.311918 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7634-140077526542768" Jul 20 21:51:46.318514 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:46.318491 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7634-140077526542768"] Jul 20 21:51:46.365898 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:46.365854 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkzfb\" (UniqueName: \"kubernetes.io/projected/a86ec73f-d76e-4ccf-825c-fa426990872b-kube-api-access-pkzfb\") pod \"test-curl-7634-140077526542768\" (UID: \"a86ec73f-d76e-4ccf-825c-fa426990872b\") " pod="opendatahub/test-curl-7634-140077526542768" Jul 20 21:51:46.466885 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:46.466847 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkzfb\" (UniqueName: \"kubernetes.io/projected/a86ec73f-d76e-4ccf-825c-fa426990872b-kube-api-access-pkzfb\") pod \"test-curl-7634-140077526542768\" (UID: \"a86ec73f-d76e-4ccf-825c-fa426990872b\") " pod="opendatahub/test-curl-7634-140077526542768" Jul 20 21:51:46.475686 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:46.475661 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkzfb\" (UniqueName: \"kubernetes.io/projected/a86ec73f-d76e-4ccf-825c-fa426990872b-kube-api-access-pkzfb\") pod \"test-curl-7634-140077526542768\" (UID: \"a86ec73f-d76e-4ccf-825c-fa426990872b\") " pod="opendatahub/test-curl-7634-140077526542768" Jul 20 21:51:46.621997 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:46.621913 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7634-140077526542768" Jul 20 21:51:46.756663 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:46.756640 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7634-140077526542768"] Jul 20 21:51:46.758778 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:51:46.758748 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda86ec73f_d76e_4ccf_825c_fa426990872b.slice/crio-7b53bd52f3b343e325fed67bf52195ce92b56440da9ab3f0c436a0d507e61734 WatchSource:0}: Error finding container 7b53bd52f3b343e325fed67bf52195ce92b56440da9ab3f0c436a0d507e61734: Status 404 returned error can't find the container with id 7b53bd52f3b343e325fed67bf52195ce92b56440da9ab3f0c436a0d507e61734 Jul 20 21:51:47.006952 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:47.006872 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7634-140077526541360" Jul 20 21:51:47.006952 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:47.006919 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8fa6fd6d3a8a3d8919221ea6f46bb6e1a3862827fbe4c6da6008219150a9b28" Jul 20 21:51:47.008245 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:47.008213 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7634-140077526542768" event={"ID":"a86ec73f-d76e-4ccf-825c-fa426990872b","Type":"ContainerStarted","Data":"7b53bd52f3b343e325fed67bf52195ce92b56440da9ab3f0c436a0d507e61734"} Jul 20 21:51:47.020486 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:51:47.020460 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7634-140077526541360\" is forbidden: User \"system:node:ip-10-0-136-32.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-32.ec2.internal' and this object" podUID="01c1f8e6-3559-4ef8-adbb-94617120f3bc" pod="opendatahub/test-curl-7634-140077526541360" Jul 20 21:51:47.270862 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:47.270832 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01c1f8e6-3559-4ef8-adbb-94617120f3bc" path="/var/lib/kubelet/pods/01c1f8e6-3559-4ef8-adbb-94617120f3bc/volumes" Jul 20 21:51:48.014258 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:48.014219 2562 generic.go:356] "Generic (PLEG): container finished" podID="a86ec73f-d76e-4ccf-825c-fa426990872b" containerID="358c94441f0bd6a340782acfe5a8cb6b45e230c2e9c90f98a5481a9709ef1f74" exitCode=0 Jul 20 21:51:48.014440 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:48.014301 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7634-140077526542768" event={"ID":"a86ec73f-d76e-4ccf-825c-fa426990872b","Type":"ContainerDied","Data":"358c94441f0bd6a340782acfe5a8cb6b45e230c2e9c90f98a5481a9709ef1f74"} Jul 20 21:51:49.153982 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:49.153960 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7634-140077526542768" Jul 20 21:51:49.175913 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:49.175876 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7634-140077526542768_a86ec73f-d76e-4ccf-825c-fa426990872b/test-curl-7634-140077526542768/0.log" Jul 20 21:51:49.189412 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:49.189387 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkzfb\" (UniqueName: \"kubernetes.io/projected/a86ec73f-d76e-4ccf-825c-fa426990872b-kube-api-access-pkzfb\") pod \"a86ec73f-d76e-4ccf-825c-fa426990872b\" (UID: \"a86ec73f-d76e-4ccf-825c-fa426990872b\") " Jul 20 21:51:49.191387 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:49.191366 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a86ec73f-d76e-4ccf-825c-fa426990872b-kube-api-access-pkzfb" (OuterVolumeSpecName: "kube-api-access-pkzfb") pod "a86ec73f-d76e-4ccf-825c-fa426990872b" (UID: "a86ec73f-d76e-4ccf-825c-fa426990872b"). InnerVolumeSpecName "kube-api-access-pkzfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 21:51:49.201122 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:49.201094 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7634-140077526542768"] Jul 20 21:51:49.204634 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:49.204615 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7634-140077526542768"] Jul 20 21:51:49.270838 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:49.270767 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a86ec73f-d76e-4ccf-825c-fa426990872b" path="/var/lib/kubelet/pods/a86ec73f-d76e-4ccf-825c-fa426990872b/volumes" Jul 20 21:51:49.291104 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:49.291076 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkzfb\" (UniqueName: \"kubernetes.io/projected/a86ec73f-d76e-4ccf-825c-fa426990872b-kube-api-access-pkzfb\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:51:49.371203 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:49.371174 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7634-140077526603504"] Jul 20 21:51:49.371558 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:49.371544 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a86ec73f-d76e-4ccf-825c-fa426990872b" containerName="test-curl-7634-140077526542768" Jul 20 21:51:49.371608 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:49.371560 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a86ec73f-d76e-4ccf-825c-fa426990872b" containerName="test-curl-7634-140077526542768" Jul 20 21:51:49.371678 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:49.371667 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a86ec73f-d76e-4ccf-825c-fa426990872b" containerName="test-curl-7634-140077526542768" Jul 20 21:51:49.374635 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:49.374621 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7634-140077526603504" Jul 20 21:51:49.382759 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:49.382737 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7634-140077526603504"] Jul 20 21:51:49.492648 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:49.492613 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql8cs\" (UniqueName: \"kubernetes.io/projected/da2c8289-3507-4c08-b6bf-00f913a3dffa-kube-api-access-ql8cs\") pod \"test-curl-7634-140077526603504\" (UID: \"da2c8289-3507-4c08-b6bf-00f913a3dffa\") " pod="opendatahub/test-curl-7634-140077526603504" Jul 20 21:51:49.594191 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:49.594099 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql8cs\" (UniqueName: \"kubernetes.io/projected/da2c8289-3507-4c08-b6bf-00f913a3dffa-kube-api-access-ql8cs\") pod \"test-curl-7634-140077526603504\" (UID: \"da2c8289-3507-4c08-b6bf-00f913a3dffa\") " pod="opendatahub/test-curl-7634-140077526603504" Jul 20 21:51:49.603010 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:49.602987 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql8cs\" (UniqueName: \"kubernetes.io/projected/da2c8289-3507-4c08-b6bf-00f913a3dffa-kube-api-access-ql8cs\") pod \"test-curl-7634-140077526603504\" (UID: \"da2c8289-3507-4c08-b6bf-00f913a3dffa\") " pod="opendatahub/test-curl-7634-140077526603504" Jul 20 21:51:49.684709 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:49.684683 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7634-140077526603504" Jul 20 21:51:49.821249 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:49.821226 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7634-140077526603504"] Jul 20 21:51:49.822543 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:51:49.822517 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda2c8289_3507_4c08_b6bf_00f913a3dffa.slice/crio-5c0219c96d13a0cfc3e3cc988e3030b564daa45f752bba80d5442d8b652bace5 WatchSource:0}: Error finding container 5c0219c96d13a0cfc3e3cc988e3030b564daa45f752bba80d5442d8b652bace5: Status 404 returned error can't find the container with id 5c0219c96d13a0cfc3e3cc988e3030b564daa45f752bba80d5442d8b652bace5 Jul 20 21:51:50.025939 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:50.025904 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7634-140077526603504" event={"ID":"da2c8289-3507-4c08-b6bf-00f913a3dffa","Type":"ContainerStarted","Data":"5c0219c96d13a0cfc3e3cc988e3030b564daa45f752bba80d5442d8b652bace5"} Jul 20 21:51:50.027089 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:50.027068 2562 scope.go:117] "RemoveContainer" containerID="358c94441f0bd6a340782acfe5a8cb6b45e230c2e9c90f98a5481a9709ef1f74" Jul 20 21:51:50.027089 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:50.027078 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7634-140077526542768" Jul 20 21:51:51.039939 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:51.039898 2562 generic.go:356] "Generic (PLEG): container finished" podID="da2c8289-3507-4c08-b6bf-00f913a3dffa" containerID="5549e7bfcd0f5b170ba008bffcdf660b2077acfad499e0cab02d2ca742b46186" exitCode=0 Jul 20 21:51:51.040411 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:51.039978 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7634-140077526603504" event={"ID":"da2c8289-3507-4c08-b6bf-00f913a3dffa","Type":"ContainerDied","Data":"5549e7bfcd0f5b170ba008bffcdf660b2077acfad499e0cab02d2ca742b46186"} Jul 20 21:51:52.179797 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:52.179765 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7634-140077526603504" Jul 20 21:51:52.205295 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:52.205266 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7634-140077526603504_da2c8289-3507-4c08-b6bf-00f913a3dffa/test-curl-7634-140077526603504/0.log" Jul 20 21:51:52.218487 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:52.218453 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ql8cs\" (UniqueName: \"kubernetes.io/projected/da2c8289-3507-4c08-b6bf-00f913a3dffa-kube-api-access-ql8cs\") pod \"da2c8289-3507-4c08-b6bf-00f913a3dffa\" (UID: \"da2c8289-3507-4c08-b6bf-00f913a3dffa\") " Jul 20 21:51:52.220440 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:52.220412 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da2c8289-3507-4c08-b6bf-00f913a3dffa-kube-api-access-ql8cs" (OuterVolumeSpecName: "kube-api-access-ql8cs") pod "da2c8289-3507-4c08-b6bf-00f913a3dffa" (UID: "da2c8289-3507-4c08-b6bf-00f913a3dffa"). InnerVolumeSpecName "kube-api-access-ql8cs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 21:51:52.237711 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:52.237686 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7634-140077526603504"] Jul 20 21:51:52.243265 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:52.243230 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7634-140077526603504"] Jul 20 21:51:52.319859 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:52.319766 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql8cs\" (UniqueName: \"kubernetes.io/projected/da2c8289-3507-4c08-b6bf-00f913a3dffa-kube-api-access-ql8cs\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:51:52.391272 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:52.391240 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7634-140077526603504"] Jul 20 21:51:52.391652 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:52.391638 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da2c8289-3507-4c08-b6bf-00f913a3dffa" containerName="test-curl-7634-140077526603504" Jul 20 21:51:52.391700 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:52.391653 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="da2c8289-3507-4c08-b6bf-00f913a3dffa" containerName="test-curl-7634-140077526603504" Jul 20 21:51:52.391752 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:52.391743 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="da2c8289-3507-4c08-b6bf-00f913a3dffa" containerName="test-curl-7634-140077526603504" Jul 20 21:51:52.394711 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:52.394695 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7634-140077526603504" Jul 20 21:51:52.403263 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:52.403240 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7634-140077526603504"] Jul 20 21:51:52.521303 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:52.521269 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46jtz\" (UniqueName: \"kubernetes.io/projected/411feb67-c277-4c50-8d7e-e01da8e1f843-kube-api-access-46jtz\") pod \"test-curl-7634-140077526603504\" (UID: \"411feb67-c277-4c50-8d7e-e01da8e1f843\") " pod="opendatahub/test-curl-7634-140077526603504" Jul 20 21:51:52.621801 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:52.621713 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46jtz\" (UniqueName: \"kubernetes.io/projected/411feb67-c277-4c50-8d7e-e01da8e1f843-kube-api-access-46jtz\") pod \"test-curl-7634-140077526603504\" (UID: \"411feb67-c277-4c50-8d7e-e01da8e1f843\") " pod="opendatahub/test-curl-7634-140077526603504" Jul 20 21:51:52.632974 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:52.632952 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46jtz\" (UniqueName: \"kubernetes.io/projected/411feb67-c277-4c50-8d7e-e01da8e1f843-kube-api-access-46jtz\") pod \"test-curl-7634-140077526603504\" (UID: \"411feb67-c277-4c50-8d7e-e01da8e1f843\") " pod="opendatahub/test-curl-7634-140077526603504" Jul 20 21:51:52.704990 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:52.704951 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7634-140077526603504" Jul 20 21:51:53.044614 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:53.044588 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7634-140077526603504"] Jul 20 21:51:53.047030 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:51:53.046979 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod411feb67_c277_4c50_8d7e_e01da8e1f843.slice/crio-d8450818f0b150a06d20cd7b4aab98630e91bc4648aaff4bfd5f1a6563a8db0b WatchSource:0}: Error finding container d8450818f0b150a06d20cd7b4aab98630e91bc4648aaff4bfd5f1a6563a8db0b: Status 404 returned error can't find the container with id d8450818f0b150a06d20cd7b4aab98630e91bc4648aaff4bfd5f1a6563a8db0b Jul 20 21:51:53.054608 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:53.054589 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7634-140077526603504" Jul 20 21:51:53.054684 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:53.054603 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c0219c96d13a0cfc3e3cc988e3030b564daa45f752bba80d5442d8b652bace5" Jul 20 21:51:53.066923 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:53.066887 2562 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7634-140077526603504" oldPodUID="da2c8289-3507-4c08-b6bf-00f913a3dffa" podUID="411feb67-c277-4c50-8d7e-e01da8e1f843" Jul 20 21:51:53.270532 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:53.270501 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da2c8289-3507-4c08-b6bf-00f913a3dffa" path="/var/lib/kubelet/pods/da2c8289-3507-4c08-b6bf-00f913a3dffa/volumes" Jul 20 21:51:54.060692 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:54.060662 2562 generic.go:356] "Generic (PLEG): container finished" podID="411feb67-c277-4c50-8d7e-e01da8e1f843" containerID="7dd16be5ac53469be26b63c84e1268b67f0c4e5f8954d85b13efd4bba47adf00" exitCode=0 Jul 20 21:51:54.060866 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:54.060704 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7634-140077526603504" event={"ID":"411feb67-c277-4c50-8d7e-e01da8e1f843","Type":"ContainerDied","Data":"7dd16be5ac53469be26b63c84e1268b67f0c4e5f8954d85b13efd4bba47adf00"} Jul 20 21:51:54.060866 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:54.060733 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7634-140077526603504" event={"ID":"411feb67-c277-4c50-8d7e-e01da8e1f843","Type":"ContainerStarted","Data":"d8450818f0b150a06d20cd7b4aab98630e91bc4648aaff4bfd5f1a6563a8db0b"} Jul 20 21:51:55.203066 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:55.203044 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7634-140077526603504" Jul 20 21:51:55.226277 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:55.226246 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7634-140077526603504_411feb67-c277-4c50-8d7e-e01da8e1f843/test-curl-7634-140077526603504/0.log" Jul 20 21:51:55.245501 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:55.245471 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46jtz\" (UniqueName: \"kubernetes.io/projected/411feb67-c277-4c50-8d7e-e01da8e1f843-kube-api-access-46jtz\") pod \"411feb67-c277-4c50-8d7e-e01da8e1f843\" (UID: \"411feb67-c277-4c50-8d7e-e01da8e1f843\") " Jul 20 21:51:55.247681 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:55.247645 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/411feb67-c277-4c50-8d7e-e01da8e1f843-kube-api-access-46jtz" (OuterVolumeSpecName: "kube-api-access-46jtz") pod "411feb67-c277-4c50-8d7e-e01da8e1f843" (UID: "411feb67-c277-4c50-8d7e-e01da8e1f843"). InnerVolumeSpecName "kube-api-access-46jtz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 21:51:55.254129 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:55.254095 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7634-140077526603504"] Jul 20 21:51:55.257575 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:55.257554 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7634-140077526603504"] Jul 20 21:51:55.269907 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:55.269877 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="411feb67-c277-4c50-8d7e-e01da8e1f843" path="/var/lib/kubelet/pods/411feb67-c277-4c50-8d7e-e01da8e1f843/volumes" Jul 20 21:51:55.346536 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:55.346449 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46jtz\" (UniqueName: \"kubernetes.io/projected/411feb67-c277-4c50-8d7e-e01da8e1f843-kube-api-access-46jtz\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:51:55.385832 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:55.385791 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7634-140077526717472"] Jul 20 21:51:55.386218 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:55.386203 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="411feb67-c277-4c50-8d7e-e01da8e1f843" containerName="test-curl-7634-140077526603504" Jul 20 21:51:55.386268 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:55.386219 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="411feb67-c277-4c50-8d7e-e01da8e1f843" containerName="test-curl-7634-140077526603504" Jul 20 21:51:55.386301 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:55.386293 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="411feb67-c277-4c50-8d7e-e01da8e1f843" containerName="test-curl-7634-140077526603504" Jul 20 21:51:55.391603 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:55.390909 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7634-140077526717472" Jul 20 21:51:55.397087 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:55.397060 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7634-140077526717472"] Jul 20 21:51:55.447511 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:55.447474 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp9ch\" (UniqueName: \"kubernetes.io/projected/2824d3a1-93d2-4adf-9339-72a9a42e30af-kube-api-access-mp9ch\") pod \"test-curl-7634-140077526717472\" (UID: \"2824d3a1-93d2-4adf-9339-72a9a42e30af\") " pod="opendatahub/test-curl-7634-140077526717472" Jul 20 21:51:55.548867 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:55.548823 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp9ch\" (UniqueName: \"kubernetes.io/projected/2824d3a1-93d2-4adf-9339-72a9a42e30af-kube-api-access-mp9ch\") pod \"test-curl-7634-140077526717472\" (UID: \"2824d3a1-93d2-4adf-9339-72a9a42e30af\") " pod="opendatahub/test-curl-7634-140077526717472" Jul 20 21:51:55.559097 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:55.559067 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp9ch\" (UniqueName: \"kubernetes.io/projected/2824d3a1-93d2-4adf-9339-72a9a42e30af-kube-api-access-mp9ch\") pod \"test-curl-7634-140077526717472\" (UID: \"2824d3a1-93d2-4adf-9339-72a9a42e30af\") " pod="opendatahub/test-curl-7634-140077526717472" Jul 20 21:51:55.703992 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:55.703913 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7634-140077526717472" Jul 20 21:51:55.835878 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:55.835853 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7634-140077526717472"] Jul 20 21:51:55.837302 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:51:55.837269 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2824d3a1_93d2_4adf_9339_72a9a42e30af.slice/crio-ae72ff2f53dded68aae795b3f75ef10b8fc185cf63bead684edb2730c56c7803 WatchSource:0}: Error finding container ae72ff2f53dded68aae795b3f75ef10b8fc185cf63bead684edb2730c56c7803: Status 404 returned error can't find the container with id ae72ff2f53dded68aae795b3f75ef10b8fc185cf63bead684edb2730c56c7803 Jul 20 21:51:56.071193 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:56.071161 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7634-140077526717472" event={"ID":"2824d3a1-93d2-4adf-9339-72a9a42e30af","Type":"ContainerStarted","Data":"ae72ff2f53dded68aae795b3f75ef10b8fc185cf63bead684edb2730c56c7803"} Jul 20 21:51:56.072415 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:56.072396 2562 scope.go:117] "RemoveContainer" containerID="7dd16be5ac53469be26b63c84e1268b67f0c4e5f8954d85b13efd4bba47adf00" Jul 20 21:51:56.072527 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:56.072422 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7634-140077526603504" Jul 20 21:51:57.079088 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:57.079057 2562 generic.go:356] "Generic (PLEG): container finished" podID="2824d3a1-93d2-4adf-9339-72a9a42e30af" containerID="ca5c3c68a2d313cc5bc2cc5c6803268cf8d899b3c1658e5eea2323d2f071e4a5" exitCode=0 Jul 20 21:51:57.079598 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:57.079167 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7634-140077526717472" event={"ID":"2824d3a1-93d2-4adf-9339-72a9a42e30af","Type":"ContainerDied","Data":"ca5c3c68a2d313cc5bc2cc5c6803268cf8d899b3c1658e5eea2323d2f071e4a5"} Jul 20 21:51:58.216965 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:58.216941 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7634-140077526717472" Jul 20 21:51:58.256944 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:58.256910 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7634-140077526717472_2824d3a1-93d2-4adf-9339-72a9a42e30af/test-curl-7634-140077526717472/0.log" Jul 20 21:51:58.275330 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:58.275295 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mp9ch\" (UniqueName: \"kubernetes.io/projected/2824d3a1-93d2-4adf-9339-72a9a42e30af-kube-api-access-mp9ch\") pod \"2824d3a1-93d2-4adf-9339-72a9a42e30af\" (UID: \"2824d3a1-93d2-4adf-9339-72a9a42e30af\") " Jul 20 21:51:58.277322 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:58.277302 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2824d3a1-93d2-4adf-9339-72a9a42e30af-kube-api-access-mp9ch" (OuterVolumeSpecName: "kube-api-access-mp9ch") pod "2824d3a1-93d2-4adf-9339-72a9a42e30af" (UID: "2824d3a1-93d2-4adf-9339-72a9a42e30af"). InnerVolumeSpecName "kube-api-access-mp9ch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 21:51:58.295614 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:58.295590 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7634-140077526717472"] Jul 20 21:51:58.297316 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:58.297297 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7634-140077526717472"] Jul 20 21:51:58.376668 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:58.376608 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp9ch\" (UniqueName: \"kubernetes.io/projected/2824d3a1-93d2-4adf-9339-72a9a42e30af-kube-api-access-mp9ch\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:51:59.091118 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:59.091085 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7634-140077526717472" Jul 20 21:51:59.091118 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:59.091117 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae72ff2f53dded68aae795b3f75ef10b8fc185cf63bead684edb2730c56c7803" Jul 20 21:51:59.107177 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:51:59.107145 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7634-140077526717472\" is forbidden: User \"system:node:ip-10-0-136-32.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-32.ec2.internal' and this object" podUID="2824d3a1-93d2-4adf-9339-72a9a42e30af" pod="opendatahub/test-curl-7634-140077526717472" Jul 20 21:51:59.270392 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:51:59.270357 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2824d3a1-93d2-4adf-9339-72a9a42e30af" path="/var/lib/kubelet/pods/2824d3a1-93d2-4adf-9339-72a9a42e30af/volumes" Jul 20 21:52:02.149385 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:02.149353 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:52:02.251604 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:02.251499 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:52:12.064026 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:12.063994 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-196c99-cf748797d-nkzxw"] Jul 20 21:52:12.064535 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:12.064207 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-196c99-cf748797d-nkzxw" podUID="6703a40c-37a5-450c-8a1f-3c181ab5c20d" containerName="maas-api" containerID="cri-o://988ab7b168a8ddc4d4faca326500350baa36e2f51c7471248558dbb9bc0f6f5a" gracePeriod=30 Jul 20 21:52:12.128499 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:12.128463 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:52:12.329014 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:12.328961 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-196c99-cf748797d-nkzxw" Jul 20 21:52:12.510175 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:12.510115 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6703a40c-37a5-450c-8a1f-3c181ab5c20d-maas-api-tls\") pod \"6703a40c-37a5-450c-8a1f-3c181ab5c20d\" (UID: \"6703a40c-37a5-450c-8a1f-3c181ab5c20d\") " Jul 20 21:52:12.510362 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:12.510215 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6703a40c-37a5-450c-8a1f-3c181ab5c20d-openshift-service-ca\") pod \"6703a40c-37a5-450c-8a1f-3c181ab5c20d\" (UID: \"6703a40c-37a5-450c-8a1f-3c181ab5c20d\") " Jul 20 21:52:12.510362 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:12.510248 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rstjr\" (UniqueName: \"kubernetes.io/projected/6703a40c-37a5-450c-8a1f-3c181ab5c20d-kube-api-access-rstjr\") pod \"6703a40c-37a5-450c-8a1f-3c181ab5c20d\" (UID: \"6703a40c-37a5-450c-8a1f-3c181ab5c20d\") " Jul 20 21:52:12.510512 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:12.510488 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6703a40c-37a5-450c-8a1f-3c181ab5c20d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6703a40c-37a5-450c-8a1f-3c181ab5c20d" (UID: "6703a40c-37a5-450c-8a1f-3c181ab5c20d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 21:52:12.512392 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:12.512362 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6703a40c-37a5-450c-8a1f-3c181ab5c20d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6703a40c-37a5-450c-8a1f-3c181ab5c20d" (UID: "6703a40c-37a5-450c-8a1f-3c181ab5c20d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 21:52:12.512533 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:12.512415 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6703a40c-37a5-450c-8a1f-3c181ab5c20d-kube-api-access-rstjr" (OuterVolumeSpecName: "kube-api-access-rstjr") pod "6703a40c-37a5-450c-8a1f-3c181ab5c20d" (UID: "6703a40c-37a5-450c-8a1f-3c181ab5c20d"). InnerVolumeSpecName "kube-api-access-rstjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 21:52:12.611342 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:12.611265 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6703a40c-37a5-450c-8a1f-3c181ab5c20d-maas-api-tls\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:52:12.611342 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:12.611292 2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6703a40c-37a5-450c-8a1f-3c181ab5c20d-openshift-service-ca\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:52:12.611342 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:12.611304 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rstjr\" (UniqueName: \"kubernetes.io/projected/6703a40c-37a5-450c-8a1f-3c181ab5c20d-kube-api-access-rstjr\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:52:13.162775 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:13.162737 2562 generic.go:356] "Generic (PLEG): container finished" podID="6703a40c-37a5-450c-8a1f-3c181ab5c20d" containerID="988ab7b168a8ddc4d4faca326500350baa36e2f51c7471248558dbb9bc0f6f5a" exitCode=0 Jul 20 21:52:13.163240 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:13.162782 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-196c99-cf748797d-nkzxw" Jul 20 21:52:13.163240 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:13.162827 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-196c99-cf748797d-nkzxw" event={"ID":"6703a40c-37a5-450c-8a1f-3c181ab5c20d","Type":"ContainerDied","Data":"988ab7b168a8ddc4d4faca326500350baa36e2f51c7471248558dbb9bc0f6f5a"} Jul 20 21:52:13.163240 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:13.162879 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-196c99-cf748797d-nkzxw" event={"ID":"6703a40c-37a5-450c-8a1f-3c181ab5c20d","Type":"ContainerDied","Data":"6f13b80637636b08428585f847965e45abc7d9b969d335aa7a4f62062fab8dea"} Jul 20 21:52:13.163240 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:13.162908 2562 scope.go:117] "RemoveContainer" containerID="988ab7b168a8ddc4d4faca326500350baa36e2f51c7471248558dbb9bc0f6f5a" Jul 20 21:52:13.172253 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:13.172233 2562 scope.go:117] "RemoveContainer" containerID="988ab7b168a8ddc4d4faca326500350baa36e2f51c7471248558dbb9bc0f6f5a" Jul 20 21:52:13.172480 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:52:13.172461 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"988ab7b168a8ddc4d4faca326500350baa36e2f51c7471248558dbb9bc0f6f5a\": container with ID starting with 988ab7b168a8ddc4d4faca326500350baa36e2f51c7471248558dbb9bc0f6f5a not found: ID does not exist" containerID="988ab7b168a8ddc4d4faca326500350baa36e2f51c7471248558dbb9bc0f6f5a" Jul 20 21:52:13.172529 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:13.172488 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"988ab7b168a8ddc4d4faca326500350baa36e2f51c7471248558dbb9bc0f6f5a"} err="failed to get container status \"988ab7b168a8ddc4d4faca326500350baa36e2f51c7471248558dbb9bc0f6f5a\": rpc error: code = NotFound desc = could not find container \"988ab7b168a8ddc4d4faca326500350baa36e2f51c7471248558dbb9bc0f6f5a\": container with ID starting with 988ab7b168a8ddc4d4faca326500350baa36e2f51c7471248558dbb9bc0f6f5a not found: ID does not exist" Jul 20 21:52:13.193817 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:13.193792 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-196c99-cf748797d-nkzxw"] Jul 20 21:52:13.195908 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:13.195886 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-196c99-cf748797d-nkzxw"] Jul 20 21:52:13.269946 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:13.269918 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6703a40c-37a5-450c-8a1f-3c181ab5c20d" path="/var/lib/kubelet/pods/6703a40c-37a5-450c-8a1f-3c181ab5c20d/volumes" Jul 20 21:52:16.636578 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:16.636539 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:52:17.166736 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:17.166695 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:52:24.451190 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:24.451158 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:52:24.533848 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:24.533808 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:52:24.731569 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:24.731475 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:52:25.012980 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:25.012953 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:52:25.914524 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:25.914490 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:52:26.046196 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:26.046158 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:52:35.959159 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:35.959100 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-046c73-68596dc579-w8hpn"] Jul 20 21:52:35.959647 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:35.959323 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-046c73-68596dc579-w8hpn" podUID="732efb04-60db-47c0-838e-5981809ad489" containerName="maas-api" containerID="cri-o://1d33432d9c059b54ca0f6093af6afffddf3dbd070fcca4dfadd4a0a6c80308a7" gracePeriod=30 Jul 20 21:52:36.067480 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:36.067448 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:52:36.210731 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:36.210670 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-046c73-68596dc579-w8hpn" Jul 20 21:52:36.279163 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:36.277238 2562 generic.go:356] "Generic (PLEG): container finished" podID="732efb04-60db-47c0-838e-5981809ad489" containerID="1d33432d9c059b54ca0f6093af6afffddf3dbd070fcca4dfadd4a0a6c80308a7" exitCode=0 Jul 20 21:52:36.279163 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:36.277300 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-046c73-68596dc579-w8hpn" event={"ID":"732efb04-60db-47c0-838e-5981809ad489","Type":"ContainerDied","Data":"1d33432d9c059b54ca0f6093af6afffddf3dbd070fcca4dfadd4a0a6c80308a7"} Jul 20 21:52:36.279163 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:36.277395 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-046c73-68596dc579-w8hpn" event={"ID":"732efb04-60db-47c0-838e-5981809ad489","Type":"ContainerDied","Data":"65fd47ce789a6b8b3784f71297c38acfa15d113e8d0518428df9d81fd0a178fc"} Jul 20 21:52:36.279163 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:36.277434 2562 scope.go:117] "RemoveContainer" containerID="1d33432d9c059b54ca0f6093af6afffddf3dbd070fcca4dfadd4a0a6c80308a7" Jul 20 21:52:36.279163 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:36.277447 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-046c73-68596dc579-w8hpn" Jul 20 21:52:36.293377 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:36.293355 2562 scope.go:117] "RemoveContainer" containerID="1d33432d9c059b54ca0f6093af6afffddf3dbd070fcca4dfadd4a0a6c80308a7" Jul 20 21:52:36.293652 ip-10-0-136-32 kubenswrapper[2562]: E0720 21:52:36.293627 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d33432d9c059b54ca0f6093af6afffddf3dbd070fcca4dfadd4a0a6c80308a7\": container with ID starting with 1d33432d9c059b54ca0f6093af6afffddf3dbd070fcca4dfadd4a0a6c80308a7 not found: ID does not exist" containerID="1d33432d9c059b54ca0f6093af6afffddf3dbd070fcca4dfadd4a0a6c80308a7" Jul 20 21:52:36.293732 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:36.293659 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d33432d9c059b54ca0f6093af6afffddf3dbd070fcca4dfadd4a0a6c80308a7"} err="failed to get container status \"1d33432d9c059b54ca0f6093af6afffddf3dbd070fcca4dfadd4a0a6c80308a7\": rpc error: code = NotFound desc = could not find container \"1d33432d9c059b54ca0f6093af6afffddf3dbd070fcca4dfadd4a0a6c80308a7\": container with ID starting with 1d33432d9c059b54ca0f6093af6afffddf3dbd070fcca4dfadd4a0a6c80308a7 not found: ID does not exist" Jul 20 21:52:36.320513 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:36.320488 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8w2c5\" (UniqueName: \"kubernetes.io/projected/732efb04-60db-47c0-838e-5981809ad489-kube-api-access-8w2c5\") pod \"732efb04-60db-47c0-838e-5981809ad489\" (UID: \"732efb04-60db-47c0-838e-5981809ad489\") " Jul 20 21:52:36.320618 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:36.320547 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/732efb04-60db-47c0-838e-5981809ad489-openshift-service-ca\") pod \"732efb04-60db-47c0-838e-5981809ad489\" (UID: \"732efb04-60db-47c0-838e-5981809ad489\") " Jul 20 21:52:36.320618 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:36.320573 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/732efb04-60db-47c0-838e-5981809ad489-maas-api-tls\") pod \"732efb04-60db-47c0-838e-5981809ad489\" (UID: \"732efb04-60db-47c0-838e-5981809ad489\") " Jul 20 21:52:36.320967 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:36.320931 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/732efb04-60db-47c0-838e-5981809ad489-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "732efb04-60db-47c0-838e-5981809ad489" (UID: "732efb04-60db-47c0-838e-5981809ad489"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 21:52:36.322417 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:36.322391 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/732efb04-60db-47c0-838e-5981809ad489-kube-api-access-8w2c5" (OuterVolumeSpecName: "kube-api-access-8w2c5") pod "732efb04-60db-47c0-838e-5981809ad489" (UID: "732efb04-60db-47c0-838e-5981809ad489"). InnerVolumeSpecName "kube-api-access-8w2c5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 21:52:36.322580 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:36.322558 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/732efb04-60db-47c0-838e-5981809ad489-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "732efb04-60db-47c0-838e-5981809ad489" (UID: "732efb04-60db-47c0-838e-5981809ad489"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 21:52:36.422247 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:36.422204 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w2c5\" (UniqueName: \"kubernetes.io/projected/732efb04-60db-47c0-838e-5981809ad489-kube-api-access-8w2c5\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:52:36.422247 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:36.422238 2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/732efb04-60db-47c0-838e-5981809ad489-openshift-service-ca\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:52:36.422247 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:36.422248 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/732efb04-60db-47c0-838e-5981809ad489-maas-api-tls\") on node \"ip-10-0-136-32.ec2.internal\" DevicePath \"\"" Jul 20 21:52:36.606128 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:36.606095 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-046c73-68596dc579-w8hpn"] Jul 20 21:52:36.618147 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:36.618113 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-046c73-68596dc579-w8hpn"] Jul 20 21:52:37.270593 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:37.270559 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="732efb04-60db-47c0-838e-5981809ad489" path="/var/lib/kubelet/pods/732efb04-60db-47c0-838e-5981809ad489/volumes" Jul 20 21:52:40.250151 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:40.250098 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:52:40.697498 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:40.697463 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:52:46.770396 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:46.770359 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:52:46.834846 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:46.834809 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:52:47.086918 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:47.086824 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:52:47.286621 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:47.286590 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:52:53.813952 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:53.813922 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-66b5c4b596-9klsp_1a9af75b-a8d9-47b3-976a-be47fbe35b4e/copy-manifests/0.log" Jul 20 21:52:53.821444 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:53.821415 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-66b5c4b596-9klsp_1a9af75b-a8d9-47b3-976a-be47fbe35b4e/manager/0.log" Jul 20 21:52:54.696392 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:54.696352 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29743065-jd5d6_7c72bde9-5a9d-40a4-9745-3d6180216ed3/cleanup/0.log" Jul 20 21:52:56.959415 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:56.959381 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-hhnpn_472af867-3e66-4dcf-a643-400f4a63a64f/manager/0.log" Jul 20 21:52:57.702703 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:57.702673 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cfcf6b877-tnp27_0bba992c-dc11-4a0e-b5bd-0a2e631bf26c/kube-auth-proxy/0.log" Jul 20 21:52:57.954295 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:57.954186 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-ktdwd_937b1c81-87b3-4cba-8e7f-7cbf6aec77b6/payload-pre-processing/0.log" Jul 20 21:52:58.723860 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:58.723828 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-hflgl_158e2721-ac21-4cd4-b344-6adbf6afdc67/main/0.log" Jul 20 21:52:58.951318 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:58.951274 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmwfzd_842a1a79-fcda-491d-a588-b364371d7648/main/0.log" Jul 20 21:52:59.186469 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:52:59.186431 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-lw2sw_cfe0744d-6ddd-473f-887b-6f1b5cc9f051/main/0.log" Jul 20 21:53:04.375550 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:04.375518 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:53:04.463192 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:04.463158 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" limit=1024 Jul 20 21:53:07.418961 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:07.418920 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2lqmn_2b3600ee-a2f5-4b18-ac22-9c868136ce69/global-pull-secret-syncer/0.log" Jul 20 21:53:07.560242 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:07.560208 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5jlkf_971c4fcc-cdb1-460e-9d0b-6eb07367f5c6/konnectivity-agent/0.log" Jul 20 21:53:07.744688 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:07.744597 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-32.ec2.internal_6ba459730c951195c7a8ffe9d5d50c7d/haproxy/0.log" Jul 20 21:53:13.128865 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:13.128827 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-hhnpn_472af867-3e66-4dcf-a643-400f4a63a64f/manager/0.log" Jul 20 21:53:15.069263 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:15.069234 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-54chl_a82a3265-df2c-4d0f-8aaf-d2434825fc10/cluster-monitoring-operator/0.log" Jul 20 21:53:15.127534 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:15.127504 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-rs8r9_700afcf1-b8cd-43d7-9c3b-e77e048bc338/kube-state-metrics/0.log" Jul 20 21:53:15.155092 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:15.155060 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-rs8r9_700afcf1-b8cd-43d7-9c3b-e77e048bc338/kube-rbac-proxy-main/0.log" Jul 20 21:53:15.198410 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:15.198381 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-rs8r9_700afcf1-b8cd-43d7-9c3b-e77e048bc338/kube-rbac-proxy-self/0.log" Jul 20 21:53:15.233979 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:15.233938 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-77c4f499cc-gvnlm_62ba6532-13ac-41d2-b6c4-0881a9123227/metrics-server/0.log" Jul 20 21:53:15.277537 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:15.277488 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-vccsj_35ab2239-07e1-4447-a1de-f3ca98dbbbc4/monitoring-plugin/0.log" Jul 20 21:53:15.589095 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:15.589062 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s45dv_2be85c96-229e-4710-b94b-1b98e7b1217c/node-exporter/0.log" Jul 20 21:53:15.626149 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:15.626033 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s45dv_2be85c96-229e-4710-b94b-1b98e7b1217c/kube-rbac-proxy/0.log" Jul 20 21:53:15.663733 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:15.663707 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s45dv_2be85c96-229e-4710-b94b-1b98e7b1217c/init-textfile/0.log" Jul 20 21:53:15.885980 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:15.885879 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_319342c3-f33c-4b9b-ac0b-c8ffe0bc312a/prometheus/0.log" Jul 20 21:53:15.926322 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:15.926287 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_319342c3-f33c-4b9b-ac0b-c8ffe0bc312a/config-reloader/0.log" Jul 20 21:53:15.964200 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:15.964175 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_319342c3-f33c-4b9b-ac0b-c8ffe0bc312a/thanos-sidecar/0.log" Jul 20 21:53:15.990020 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:15.989995 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_319342c3-f33c-4b9b-ac0b-c8ffe0bc312a/kube-rbac-proxy-web/0.log" Jul 20 21:53:16.017740 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:16.017713 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_319342c3-f33c-4b9b-ac0b-c8ffe0bc312a/kube-rbac-proxy/0.log" Jul 20 21:53:16.047061 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:16.047035 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_319342c3-f33c-4b9b-ac0b-c8ffe0bc312a/kube-rbac-proxy-thanos/0.log" Jul 20 21:53:16.073880 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:16.073854 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_319342c3-f33c-4b9b-ac0b-c8ffe0bc312a/init-config-reloader/0.log" Jul 20 21:53:16.171681 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:16.171594 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-fphgz_b8372209-79c7-49f4-89eb-c9ad3846e1ae/prometheus-operator-admission-webhook/0.log" Jul 20 21:53:16.218653 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:16.218621 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-84ddb46cc7-r6khm_a16c41e7-0bed-413b-abbb-17ef8ba56057/telemeter-client/0.log" Jul 20 21:53:16.276015 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:16.275990 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-84ddb46cc7-r6khm_a16c41e7-0bed-413b-abbb-17ef8ba56057/reload/0.log" Jul 20 21:53:16.304427 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:16.304403 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-84ddb46cc7-r6khm_a16c41e7-0bed-413b-abbb-17ef8ba56057/kube-rbac-proxy/0.log" Jul 20 21:53:16.338572 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:16.338535 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c8d56f8b8-hxvgr_ff6806f9-668b-4512-906a-df4a69b73ba9/thanos-query/0.log" Jul 20 21:53:16.371104 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:16.371077 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c8d56f8b8-hxvgr_ff6806f9-668b-4512-906a-df4a69b73ba9/kube-rbac-proxy-web/0.log" Jul 20 21:53:16.413248 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:16.413227 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c8d56f8b8-hxvgr_ff6806f9-668b-4512-906a-df4a69b73ba9/kube-rbac-proxy/0.log" Jul 20 21:53:16.437924 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:16.437840 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c8d56f8b8-hxvgr_ff6806f9-668b-4512-906a-df4a69b73ba9/prom-label-proxy/0.log" Jul 20 21:53:16.461066 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:16.461047 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c8d56f8b8-hxvgr_ff6806f9-668b-4512-906a-df4a69b73ba9/kube-rbac-proxy-rules/0.log" Jul 20 21:53:16.486858 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:16.486835 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c8d56f8b8-hxvgr_ff6806f9-668b-4512-906a-df4a69b73ba9/kube-rbac-proxy-metrics/0.log" Jul 20 21:53:17.033805 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:17.033769 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cffm9/perf-node-gather-daemonset-h9q9p"] Jul 20 21:53:17.034200 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:17.034185 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2824d3a1-93d2-4adf-9339-72a9a42e30af" containerName="test-curl-7634-140077526717472" Jul 20 21:53:17.034281 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:17.034201 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2824d3a1-93d2-4adf-9339-72a9a42e30af" containerName="test-curl-7634-140077526717472" Jul 20 21:53:17.034281 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:17.034215 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="732efb04-60db-47c0-838e-5981809ad489" containerName="maas-api" Jul 20 21:53:17.034281 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:17.034221 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="732efb04-60db-47c0-838e-5981809ad489" containerName="maas-api" Jul 20 21:53:17.034281 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:17.034242 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6703a40c-37a5-450c-8a1f-3c181ab5c20d" containerName="maas-api" Jul 20 21:53:17.034281 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:17.034247 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6703a40c-37a5-450c-8a1f-3c181ab5c20d" containerName="maas-api" Jul 20 21:53:17.034461 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:17.034306 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="732efb04-60db-47c0-838e-5981809ad489" containerName="maas-api" Jul 20 21:53:17.034461 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:17.034316 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2824d3a1-93d2-4adf-9339-72a9a42e30af" containerName="test-curl-7634-140077526717472" Jul 20 21:53:17.034461 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:17.034324 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6703a40c-37a5-450c-8a1f-3c181ab5c20d" containerName="maas-api" Jul 20 21:53:17.037360 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:17.037335 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cffm9/perf-node-gather-daemonset-h9q9p" Jul 20 21:53:17.040193 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:17.040171 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cffm9\"/\"default-dockercfg-tbtzm\"" Jul 20 21:53:17.040456 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:17.040437 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cffm9\"/\"openshift-service-ca.crt\"" Jul 20 21:53:17.041167 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:17.041151 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cffm9\"/\"kube-root-ca.crt\"" Jul 20 21:53:17.054802 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:17.054776 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cffm9/perf-node-gather-daemonset-h9q9p"] Jul 20 21:53:17.086109 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:17.086074 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-622bp\" (UniqueName: \"kubernetes.io/projected/4e4e4acb-81d3-49e0-b0de-ab130ff4e5ae-kube-api-access-622bp\") pod \"perf-node-gather-daemonset-h9q9p\" (UID: \"4e4e4acb-81d3-49e0-b0de-ab130ff4e5ae\") " pod="openshift-must-gather-cffm9/perf-node-gather-daemonset-h9q9p" Jul 20 21:53:17.086537 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:17.086175 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e4e4acb-81d3-49e0-b0de-ab130ff4e5ae-sys\") pod \"perf-node-gather-daemonset-h9q9p\" (UID: \"4e4e4acb-81d3-49e0-b0de-ab130ff4e5ae\") " pod="openshift-must-gather-cffm9/perf-node-gather-daemonset-h9q9p" Jul 20 21:53:17.086537 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:17.086213 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4e4e4acb-81d3-49e0-b0de-ab130ff4e5ae-lib-modules\") pod \"perf-node-gather-daemonset-h9q9p\" (UID: \"4e4e4acb-81d3-49e0-b0de-ab130ff4e5ae\") " pod="openshift-must-gather-cffm9/perf-node-gather-daemonset-h9q9p" Jul 20 21:53:17.086537 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:17.086294 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4e4e4acb-81d3-49e0-b0de-ab130ff4e5ae-podres\") pod \"perf-node-gather-daemonset-h9q9p\" (UID: \"4e4e4acb-81d3-49e0-b0de-ab130ff4e5ae\") " pod="openshift-must-gather-cffm9/perf-node-gather-daemonset-h9q9p" Jul 20 21:53:17.086537 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:17.086327 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4e4e4acb-81d3-49e0-b0de-ab130ff4e5ae-proc\") pod \"perf-node-gather-daemonset-h9q9p\" (UID: \"4e4e4acb-81d3-49e0-b0de-ab130ff4e5ae\") " pod="openshift-must-gather-cffm9/perf-node-gather-daemonset-h9q9p" Jul 20 21:53:17.187337 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:17.187301 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4e4e4acb-81d3-49e0-b0de-ab130ff4e5ae-lib-modules\") pod \"perf-node-gather-daemonset-h9q9p\" (UID: \"4e4e4acb-81d3-49e0-b0de-ab130ff4e5ae\") " pod="openshift-must-gather-cffm9/perf-node-gather-daemonset-h9q9p" Jul 20 21:53:17.187540 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:17.187368 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4e4e4acb-81d3-49e0-b0de-ab130ff4e5ae-podres\") pod \"perf-node-gather-daemonset-h9q9p\" (UID: \"4e4e4acb-81d3-49e0-b0de-ab130ff4e5ae\") " pod="openshift-must-gather-cffm9/perf-node-gather-daemonset-h9q9p" Jul 20 21:53:17.187540 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:17.187394 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4e4e4acb-81d3-49e0-b0de-ab130ff4e5ae-proc\") pod \"perf-node-gather-daemonset-h9q9p\" (UID: \"4e4e4acb-81d3-49e0-b0de-ab130ff4e5ae\") " pod="openshift-must-gather-cffm9/perf-node-gather-daemonset-h9q9p" Jul 20 21:53:17.187540 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:17.187416 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-622bp\" (UniqueName: \"kubernetes.io/projected/4e4e4acb-81d3-49e0-b0de-ab130ff4e5ae-kube-api-access-622bp\") pod \"perf-node-gather-daemonset-h9q9p\" (UID: \"4e4e4acb-81d3-49e0-b0de-ab130ff4e5ae\") " pod="openshift-must-gather-cffm9/perf-node-gather-daemonset-h9q9p" Jul 20 21:53:17.187540 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:17.187479 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4e4e4acb-81d3-49e0-b0de-ab130ff4e5ae-lib-modules\") pod \"perf-node-gather-daemonset-h9q9p\" (UID: \"4e4e4acb-81d3-49e0-b0de-ab130ff4e5ae\") " pod="openshift-must-gather-cffm9/perf-node-gather-daemonset-h9q9p" Jul 20 21:53:17.187540 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:17.187488 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e4e4acb-81d3-49e0-b0de-ab130ff4e5ae-sys\") pod \"perf-node-gather-daemonset-h9q9p\" (UID: \"4e4e4acb-81d3-49e0-b0de-ab130ff4e5ae\") " pod="openshift-must-gather-cffm9/perf-node-gather-daemonset-h9q9p" Jul 20 21:53:17.187540 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:17.187516 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4e4e4acb-81d3-49e0-b0de-ab130ff4e5ae-proc\") pod \"perf-node-gather-daemonset-h9q9p\" (UID: \"4e4e4acb-81d3-49e0-b0de-ab130ff4e5ae\") " pod="openshift-must-gather-cffm9/perf-node-gather-daemonset-h9q9p" Jul 20 21:53:17.187540 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:17.187529 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e4e4acb-81d3-49e0-b0de-ab130ff4e5ae-sys\") pod \"perf-node-gather-daemonset-h9q9p\" (UID: \"4e4e4acb-81d3-49e0-b0de-ab130ff4e5ae\") " pod="openshift-must-gather-cffm9/perf-node-gather-daemonset-h9q9p" Jul 20 21:53:17.187540 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:17.187540 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4e4e4acb-81d3-49e0-b0de-ab130ff4e5ae-podres\") pod \"perf-node-gather-daemonset-h9q9p\" (UID: \"4e4e4acb-81d3-49e0-b0de-ab130ff4e5ae\") " pod="openshift-must-gather-cffm9/perf-node-gather-daemonset-h9q9p" Jul 20 21:53:17.196639 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:17.196612 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-622bp\" (UniqueName: \"kubernetes.io/projected/4e4e4acb-81d3-49e0-b0de-ab130ff4e5ae-kube-api-access-622bp\") pod \"perf-node-gather-daemonset-h9q9p\" (UID: \"4e4e4acb-81d3-49e0-b0de-ab130ff4e5ae\") " pod="openshift-must-gather-cffm9/perf-node-gather-daemonset-h9q9p" Jul 20 21:53:17.348158 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:17.348047 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cffm9/perf-node-gather-daemonset-h9q9p" Jul 20 21:53:17.485213 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:17.485183 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cffm9/perf-node-gather-daemonset-h9q9p"] Jul 20 21:53:17.488359 ip-10-0-136-32 kubenswrapper[2562]: W0720 21:53:17.488328 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4e4e4acb_81d3_49e0_b0de_ab130ff4e5ae.slice/crio-7c92efdc95c81539f8b98d9425dc7b8a3d42cbd4a0aa8095b7721dc3e5b648be WatchSource:0}: Error finding container 7c92efdc95c81539f8b98d9425dc7b8a3d42cbd4a0aa8095b7721dc3e5b648be: Status 404 returned error can't find the container with id 7c92efdc95c81539f8b98d9425dc7b8a3d42cbd4a0aa8095b7721dc3e5b648be Jul 20 21:53:18.486823 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:18.486791 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cffm9/perf-node-gather-daemonset-h9q9p" event={"ID":"4e4e4acb-81d3-49e0-b0de-ab130ff4e5ae","Type":"ContainerStarted","Data":"850568a8eb9bc90efaebf9312c9aba220feb424d3d8dafd2e3379e286a55d9a0"} Jul 20 21:53:18.487312 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:18.486842 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cffm9/perf-node-gather-daemonset-h9q9p" event={"ID":"4e4e4acb-81d3-49e0-b0de-ab130ff4e5ae","Type":"ContainerStarted","Data":"7c92efdc95c81539f8b98d9425dc7b8a3d42cbd4a0aa8095b7721dc3e5b648be"} Jul 20 21:53:18.487312 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:18.486919 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cffm9/perf-node-gather-daemonset-h9q9p" Jul 20 21:53:18.506321 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:18.506272 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-cffm9/perf-node-gather-daemonset-h9q9p" podStartSLOduration=1.506257819 podStartE2EDuration="1.506257819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 21:53:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 21:53:18.503761908 +0000 UTC m=+3037.829464854" watchObservedRunningTime="2026-07-20 21:53:18.506257819 +0000 UTC m=+3037.831960762" Jul 20 21:53:18.593660 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:18.593630 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6bd98d7844-x6slf_f498ba96-6fee-4314-a284-af05c8147096/console/0.log" Jul 20 21:53:19.196652 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:19.196620 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-nft7r_5ef2d9b0-d68d-417e-bc1e-3fcd873aa850/volume-data-source-validator/0.log" Jul 20 21:53:20.336466 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:20.336438 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zp575_7cc5fdb7-149c-47f7-805a-cadc7c5f2599/dns/0.log" Jul 20 21:53:20.355757 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:20.355718 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zp575_7cc5fdb7-149c-47f7-805a-cadc7c5f2599/kube-rbac-proxy/0.log" Jul 20 21:53:20.402704 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:20.402684 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qcsfr_dde8ddcb-7c86-4c9d-bd00-11129c8ca35e/dns-node-resolver/0.log" Jul 20 21:53:21.048717 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:21.048688 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lgq8n_ef68231f-0506-4450-9fcb-bc14af3b7d76/node-ca/0.log" Jul 20 21:53:22.200989 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:22.200958 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cfcf6b877-tnp27_0bba992c-dc11-4a0e-b5bd-0a2e631bf26c/kube-auth-proxy/0.log" Jul 20 21:53:22.318047 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:22.318018 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-ktdwd_937b1c81-87b3-4cba-8e7f-7cbf6aec77b6/payload-pre-processing/0.log" Jul 20 21:53:23.163978 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:23.163952 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jd2kv_d1bb6ff3-c7d2-4be7-9e62-983c4f6bbff4/serve-healthcheck-canary/0.log" Jul 20 21:53:23.793889 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:23.793854 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rq8g2_399a1523-41ed-4080-9321-66533820eded/kube-rbac-proxy/0.log" Jul 20 21:53:23.812457 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:23.812431 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rq8g2_399a1523-41ed-4080-9321-66533820eded/exporter/0.log" Jul 20 21:53:23.835068 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:23.835042 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rq8g2_399a1523-41ed-4080-9321-66533820eded/extractor/0.log" Jul 20 21:53:24.502123 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:24.502094 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cffm9/perf-node-gather-daemonset-h9q9p" Jul 20 21:53:26.827924 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:26.827900 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-66b5c4b596-9klsp_1a9af75b-a8d9-47b3-976a-be47fbe35b4e/manager/0.log" Jul 20 21:53:26.849584 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:26.849547 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-66b5c4b596-9klsp_1a9af75b-a8d9-47b3-976a-be47fbe35b4e/copy-manifests/0.log" Jul 20 21:53:27.481012 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:27.480981 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cfcf6b877-tnp27_0bba992c-dc11-4a0e-b5bd-0a2e631bf26c/kube-auth-proxy/0.log" Jul 20 21:53:27.564147 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:27.564099 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-ktdwd_937b1c81-87b3-4cba-8e7f-7cbf6aec77b6/payload-pre-processing/0.log" Jul 20 21:53:28.232561 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:28.232529 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7475cd49b4-ns279_b047db61-e3ec-4c54-83f9-d14bc3dd38f5/manager/0.log" Jul 20 21:53:28.938612 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:28.938571 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cfcf6b877-tnp27_0bba992c-dc11-4a0e-b5bd-0a2e631bf26c/kube-auth-proxy/0.log" Jul 20 21:53:29.016677 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:29.016643 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-ktdwd_937b1c81-87b3-4cba-8e7f-7cbf6aec77b6/payload-pre-processing/0.log" Jul 20 21:53:33.001859 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:33.001830 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-59hpz_b3db7a22-0821-4a68-bc79-1efbd85b5483/migrator/0.log" Jul 20 21:53:33.021255 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:33.021227 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-59hpz_b3db7a22-0821-4a68-bc79-1efbd85b5483/graceful-termination/0.log" Jul 20 21:53:33.359576 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:33.359530 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-lv5br_a002a221-d9f5-42b1-b07b-4b6e946cf5c5/kube-storage-version-migrator-operator/1.log" Jul 20 21:53:33.361811 ip-10-0-136-32 kubenswrapper[2562]: I0720 21:53:33.361770 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-lv5br_a002a221-d9f5-42b1-b07b-4b6e946cf5c5/kube-storage-version-migrator-operator/0.log"